Migrate GitLab Instance to new Host

Here I migrate a GitLab installation from one server to another. The hostname and IP addresses will be different. Both were installed from the Omnibus version. However, currently the exact version numbers differ.

In order to migrate successfully, both versions must be exactly the same type and version. For example, both installed using the Omnibus version and exactly the same version number.

If you are unsure whether you installed the Omnibus version, you can confirm with the below command. This will fail if not. If it returns output similar to the below, you have the Omnibus version. It also shows me that currently have version 10.2.3 installed.

root@repo:~# gitlab-rake gitlab:env:info

System information
System:         Ubuntu 16.04
Current User:   git
Using RVM:      no
Ruby Version:   2.3.5p376
Gem Version:    2.6.13
Bundler Version:1.13.7
Rake Version:   12.1.0
Redis Version:  3.2.5
Git Version:    2.13.6
Sidekiq Version:5.0.4
Go Version:     unknown

GitLab information
Version:        10.2.3
Revision:       3141105
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://gitlab.openit-uk.com
HTTP Clone URL: https://gitlab.openit-uk.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.openit-uk.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: no

GitLab Shell
Version:        5.9.4
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks
Git:            /opt/gitlab/embedded/bin/git

If you do not have the Omnibus version, consult the official documentation on how to migrate. You need to make sure the version you are going to and from are the same.

The Strategy

  1. Install GitLab on new server [10.7.3]
  2. Update old GitLab to the latest version [10.7.3]
  3. Backup old GitLab
  4. Transfer backup files to new host
  5. Restore from backup on new instance

Install GitLab

Install GitLab on the new server.

Update GitLab

This can simply be achieved by updating the system.

apt-get update
apt-get dist-upgrade

GitLab should now be updated to the latest version. It should also have created an application backup in the below location as part of the process.

root@repo:~# ls -l /var/opt/gitlab/backups/
total 204
-rw------- 1 git git  71680 May 16 19:29 1526498956_2018_05_16_10.2.3_gitlab_backup.tar

Check the version is now the latest and the same as the origin server.

root@repo:~# gitlab-rake gitlab:env:info | grep "GitLab information" -A1
GitLab information
Version:	10.7.3

Backup GitLab

To fully backup, you need to take both a configuration and a application backup. The configuration backup includes all of GitLabs configuration including the external URL. The application backup includes all of your repositories.

Configuration Backup

All GitLab configuration is inside /etc/gitlab. To backup this directory, run:

sudo sh -c 'umask 0077; tar -cf $(date "+etc-gitlab-%s.tar") -C / etc/gitlab'

This creates the following file in the current working directory.

root@repo:~# ls -la etc-gitlab-1526500078.tar
-rw------- 1 root root 102400 May 16 19:47 etc-gitlab-1526500078.tar

With the following content.

root@repo:~# tar -tvf etc-gitlab-1526500078.tar
drwxrwxr-x root/root         0 2017-12-09 23:32 etc/gitlab/
drwxr-xr-x root/root         0 2017-12-06 14:58 etc/gitlab/trusted-certs/
drwx------ root/root         0 2017-12-09 23:41 etc/gitlab/ssl/
-rw-r--r-- root/root      2025 2017-12-09 23:09 etc/gitlab/ssl/repo.pikedom.com.crt
-rw-r--r-- root/root      1708 2017-12-09 23:08 etc/gitlab/ssl/repo.pikedom.com.key
-rw-r--r-- root/root      2025 2017-12-09 23:39 etc/gitlab/ssl/gitlab.openit-uk.com.crt
-rw-r--r-- root/root      1708 2017-12-09 23:41 etc/gitlab/ssl/gitlab.openit-uk.com.key
-rw------- root/root     14906 2018-05-16 19:31 etc/gitlab/gitlab-secrets.json
-rw------- root/root     72812 2017-12-09 23:32 etc/gitlab/gitlab.rb

Application Backup

By default, backups are kept in /var/opt/gitlab/backups.

root@repo:~# grep backup_path /etc/gitlab/gitlab.rb
# gitlab_rails['manage_backup_path'] = true
# gitlab_rails['backup_path'] = "/var/opt/gitlab/backups"

If you change this location, be sure to run the below after.

sudo gitlab-ctl reconfigure

To actually take the application backup, run the following.

sudo gitlab-rake gitlab:backup:create

This creates a backup in the following location.

root@repo:~# ls -ltr /var/opt/gitlab/backups/
total 204
-rw------- 1 git git  71680 May 16 19:29 1526498956_2018_05_16_10.2.3_gitlab_backup.tar
-rw------- 1 git git 133120 May 16 20:14 1526501659_2018_05_16_10.7.3_gitlab_backup.tar

Host Backup (optional)

If rebuilding the machine and keeping the same IP, to avoid having to delete the host key entry in the ~/.ssh/know_hosts file, run the following to backup the SSH host keys.

sudo tar -cvf $(date "+hostkeys-%s.tar") $(grep HostKey /etc/ssh/sshd_config | grep -v ^'#' | awk '{print $2}')


Lets zip and compress these files into a single archive for transfer. Although I don’t plan on restoring the host keys, here I zip and compress all three all the same.


Zip and compress with the following.

sudo tar -cvzf ~/gitlab-backup.tar.gz /var/opt/gitlab/backups/1526501659_2018_05_16_10.7.3_gitlab_backup.tar /root/etc-gitlab-1526500078.tar /root/hostkeys-1526501148.tar

Use scp to copy the backup securely over the internet. Here I run the below from the new GitLab server to pull the above backup to the current working directory.

sudo scp -v root@old.dummydomains.org.uk:/root/gitlab-backup.tar.gz .


As I am mainly concerned about the Git repositories themselves, I will only be doing an application restore.

If you are moving to a new host with the same hostname etc., you should also restore the configuration backup to /etc/gitlab/. If you want everything to be the same but the hostname is different, you can manually amend anything such as external_url before running sudo gitlab-ctl reconfigure.

Application Backup Restore

Stop processes using the database.

sudo gitlab-ctl stop unicorn
sudo gitlab-ctl stop sidekiq
sudo gitlab-ctl status

Check they’re actually down.

sudo gitlab-ctl status | grep ^down

Move the application backup to the backups directory as defined in gitlab.rb and correct the owner if necessary.

sudo mv -v ~/temp/1526501659_2018_05_16_10.7.3_gitlab_backup.tar /var/opt/gitlab/backups/
sudo chown -v git:git /var/opt/gitlab/backups/1526501659_2018_05_16_10.7.3_gitlab_backup.tar

Then restore.

sudo gitlab-rake gitlab:backup:restore BACKUP=1526501659_2018_05_16_10.7.3

Now restart GitLab.

sudo gitlab-ctl start

And check for problems.

sudo gitlab-rake gitlab:check SANITIZE=true

If all was successful, you should be able to log into the new site as before with whatever users you had setup. If you had 2FA enabled, you may need to restore the gitlab-secrets.json.

Configuration Backup Restore

Exactly what you choose to restore from the /etc/gitlab/ directory is up to you. None maybe necessary. If any of your users had 2FA setup, you will need to restore the gitlab-secrects.json file or else they will lose their access. If unsure, you could always do a diff to compare the differences.

diff old-gitlab.rb new-gitlab.rb

If moving to a new host but the hostname and everything should remain the same, you can simply move the old directory out the way and replace with the old.

First, stop unicorn and sidekiq at a minimum.

sudo gitlab-ctl stop

Then restore the configuration directory.

mv -v /etc/gitlab{,.original}
mv -v ~/temp/etc/gitlab /etc/gitlab

Then start GitLab again and check.

sudo gitlab-ctl start
sudo gitlab-rake gitlab:check SANITIZE=true

Fix any issue and then re-run if necessary. Try re-configuring if you have issues.

sudo gitlab-ctl reconfigure

If even the IP address remains the same as before and you want to prevent your users from seeing an SSH man-in-the-middle attack warning, you will also need to restore the HostKeys.

You need to replace the HostKeys defined in your sshd_config.

andy@repo:~$ grep HostKey /etc/ssh/sshd_config 
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

…with the ones from your backup.

andy@repo:~$ tar -tvf temp/hostkeys-1526501148.tar.gz 
-rw------- root/root      1679 2017-12-05 13:37 etc/ssh/ssh_host_rsa_key
-rw------- root/root       672 2017-12-05 13:37 etc/ssh/ssh_host_dsa_key
-rw------- root/root       227 2017-12-05 13:37 etc/ssh/ssh_host_ecdsa_key
-rw------- root/root       399 2017-12-05 13:37 etc/ssh/ssh_host_ed25519_key

Rename the existing keys.

for i in $(grep ^HostKey /etc/ssh/sshd_config | awk '{print $2}'); do sudo mv -v $i $i.bkp-$(date +%s);done

Then restore the old ones in-place.

sudo tar -xvzf temp/hostkeys-1526501148.tar.gz -C /

Restart GitLab and check for errors.

sudo gitlab-ctl restart
sudo gitlab-rake gitlab:check SANITIZE=true



Install and Configure GitLab on Ubuntu 16.04


Run to the below command to install the GitLab dependencies.

sudo apt-get update
sudo apt-get install ca-certificates curl openssh-server postfix

When prompted, select Internet Site when configuring Postfix.

Postfix - Mail Type
Postfix – Mail Type

Enter the FQDN.

Postfix - enter FQDN
Postfix – enter FQDN

Installing GitLab

GitLab provides a bash script that add their repository for you. Just download it and run it.

cd /tmp
curl -LO https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh
sudo bash /tmp/script.deb.sh

Now you can install the GitLab package using the apt-get package manager.

sudo apt-get install gitlab-ce

Configuring GitLab

You need to first edit the configuration file.

sudo vim /etc/gitlab/gitlab.rb

I added the following line to define my primary hostname and URL.

external_url 'https://repo.dummydomains.org.uk'

I also enabled letsencrypt. If you do, make sure the DNS is setup already.

letsencrypt['enable'] = true
letsencrypt['contact_emails'] = ['postmaster@dummydomains.org.uk']
letsencrypt['auto_renew'] = true
letsencrypt['auto_renew_hour'] = "3"
letsencrypt['auto_renew_minute'] = "30"
letsencrypt['auto_renew_day_of_month'] = "*/7"

This will auto-renew at 3:30 AM every seventh day. Now run the below command to bring about your changes.

sudo gitlab-ctl reconfigure

If all was successful, you should see gitlab Reconfigured! on the last line of output.

Browse to the URL and change your password.

GitLab - Change Password
GitLab – Change Password

Then log in with the root user and the password you just created.

GitLab - Sign In
GitLab – Sign In

GitLab is now installed!

GitLab - Sign In
GitLab – Sign In

It’s time to start configuring it! You may want to look at preventing people signing up.



The Quick Start Guide to vcsh/mr Setup

I recently wrote a post about how to install configure vcsh and mr to convienently manage and use all your repositories.  However for demostration this was quite a manual and long process.  To achieve a very similar outcome in a much shorter time, you could use the provided template and customise from there.

The below will use vcsh to clone the git repository to mr.git in the ~/.config/vcsh/repo.d/ directory.

$ yaourt -Sy vcsh myrepos
$ vcsh clone https://github.com/RichiH/vcsh_mr_template mr
$ mr update

The above repository is not owned by me so I can not push changes to it. As such you will need to edit the configuration and point it to your repository. I have setup the follow:


Edit ~/.config/mr/available.d/mr.vcsh like so.

$ vim ~/.config/mr/available.d/mr.vcsh

Don’t forget to replace the remote repository with your own.

checkout = vcsh clone https://github.com/crazzyfool/mr.git mr
update = vcsh mr pull
push =  vcsh mr push
status = vcsh mr status
gc = vcsh mr gc

To see the current remote repository.

[andy@home-pc ~]$ vcsh mr remote show origin
* remote origin
  Fetch URL: https://github.com/RichiH/vcsh_mr_template
  Push  URL: https://github.com/RichiH/vcsh_mr_template
  HEAD branch: master
  Remote branch:
    master tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (up to date)

To remove the above origin from the repository and set your own as the upstream repository.

$ vcsh mr remote remove origin
$ vcsh mr remote add origin https://github.com/crazzyfool/mr.git
$ vcsh mr push --set-upstream origin master

Now you can add configurations as you like and push them to your remote repository simply with.

$ vcsh mr push

Or you could of course use mr to push changes to all repositories.

$ mr push

But that’s it! Take a look at my previous article or the official documentation for more information about how to add repositories.

Managing dot files with vcsh and myrepo

Ever wanted to manage all those ~/.conf files with Git?  Probably not but if you know anything about Git, the advantages are obvious. Is it even possible though you may ask!?! To do so would mean separating the working tree from the actual git repository. Fortunately Git allows for just that.

By default, when you create a non-bare repository, you end up with the actual repository, a .git directory being created in the directory you want to track. The directory you want to track is actually called the working tree. The repository itself is a load of files (objects really) that maps and tracks the state of a working tree. That’s what a non-bare repository is in fact – a git repository with a checked-out working tree included. Fortunately the working tree and the git repository do not actually need to be in the same location. You can set these with the git variables, $GIT_DIR for the repository location and $GIT_WORK_TREE for the working tree.

Best of all however is that you don’t even need to get your hands dirty with this kind of thing as a nice chap called Richard Hartmann has taken all the potential pain out of it with vcsh!  All you need is a basic/working knowledge of Git.

The Plan

In this example I want to track configuration changes to two applications, the i3 Window Manager and Bash.  I’ll walk you through how to use vcsh for this, and then demonstrate how a program called mr, available with the myrepos package, can make your life even simpler.

You’ll need to setup a separate repository for each application configuration you want to track. Here I have setup one for i3wm and bash.


Later you’ll also need one for mr.



Using vcsh allows one to have multiple working trees in the same directory by storing the actual repository data in a separate location.


I run Manjaro so to install its:

$ yaourt -Sy vcsh

Add an Application Configuration

So I want to use vcsh to manage the configuration changes to the i3 Windows Manager, located ~/.i3/config.

vcsh init config-i3
vcsh config-i3 add ~/.i3/
vcsh config-i3 commit -m "My initial i3 config"
vcsh config-i3 remote add origin https://github.com/crazzyfool/config-i3.git
vcsh config-i3 push -u origin master

You may also want to create a gitignore file too.

vcsh write-gitignore config-i3

See below for more about that.

Add a Second Application Configuration

vcsh init config-bash
vcsh config-bash add ~/.bashrc
vcsh config-bash commit -m "My initial bash config"
vcsh config-bash remote add origin https://github.com/crazzyfool/config-bash.git
vcsh config-bash push -u origin master

Maybe you want to add a second file to the bash configuration.

vcsh config-bash add ~/.bash_profile
vcsh config-bash commit -m "Added .bash_profile"
vcsh config-bash push

What Just Happened?

All the actual local repositories are stored in ~/.config/vcsh/repo.d while the working trees are located in the ~/ directory.

└── repo.d
    ├── config-bash.git
    └── config-i3.git

Using vcsh to Manage your Configuration

You can get a full list of options from the help menu.

$ vcsh help
usage: vcsh<options> <command>

   -c <file>            Source file
   -d                   Enable debug mode
   -v                   Enable verbose mode

   clone [-b <branch>] \
         <remote> \
         [<repo>]       Clone from an existing repository
   commit               Commit in all repositories
   delete <repo>        Delete an existing repository
   enter <repo>         Enter repository; spawn new instance of $SHELL
   foreach [<-g>]
     <git command>      Execute a command for every repository
   help                 Display this help text
   init <repo>          Initialize a new repository
   list                 List all repositories
   list-tracked \
        [<repo>]        List all files tracked all or one repositories
   list-untracked \
        [<-a>] [<-r>]
        [<repo>]        List all files not tracked by all or one repositories
   pull                 Pull from all vcsh remotes
   push                 Push to vcsh remotes
   rename <repo> \
          <newname>     Rename repository
   run <repo> \
       <command>        Use this repository
   status \
     [--terse] [<repo>] Show statuses of all/one vcsh repositories
   upgrade <repo>       Upgrade repository to currently recommended settings
   version              Print version information
   which <substring>    Find substring in name of any tracked file
   write-gitignore \
   <repo>               Write .gitignore.d/<repo> via git ls-files

   <repo> <git command> Shortcut to run git commands directly
   <repo>               Shortcut to enter repository

Using vcsh to List

You can list all repositories with vcsh list.

$ vcsh list

To list all the currently tracked files.

$ vcsh list-tracked

If you just want to see what files one particular repository, such as config-bash is tracking, you can use the following.

$ vcsh list-tracked config-bash

Editing Configuration

Once you’ve edited a file, say .bashrc, you will need to add and commit those changes. Running the below command will show you that file has been modified.

$ vcsh config-bash status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   .bashrc

no changes added to commit (use "git add" and/or "git commit -a")

If you haven’t already created a .gitignore file, your output might look more verbose.

[andy@home-pc ~]$ vcsh config-bash status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   .bashrc

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	.moonchild productions/

no changes added to commit (use "git add" and/or "git commit -a")

If this is the case, you can get vcsh to create one for you.

$ vcsh write-gitignore config-bash

Now the ~/.gitignore.d/config-bash file will exist with the following content:

$ cat ~/.gitignore.d/config-bash 

To add that change commit.

$ vcsh config-bash add .bashrc
$ vcsh config-bash commit -m "Added ll alias to .bashrc"
$ vcsh config-bash push


You can set the status of a repository with.

$ vcsh config-bash status

Or the status of all repositories with.

$ vcsh status

Pull Changes

To pull changes from Github.

$ vcsh pull

…or for just one repository.

$ vcsh config-i3 pull

Push Changes

To push your changes back up to Github.

$ vcsh push

Or for a given repository.

$ vcsh config-i3 push

Running Git Commands

You can also just run normal Git command appended onto vcsh repo-name. For example, git log.

$ vcsh config-bash log
commit bd775543cbaeca89c20b5dad421f15b1cb02da45 (HEAD -> master, origin/master)
Author: Andrew Pike <hostmaster@pikedom.com>
Date:   Sat Apr 28 20:35:44 2018 +0100

    Added ll alias to .bashrc

commit 5c9f459ed14de5e677e02f9eff7d3c749ddb07c8
Author: Andrew Pike <hostmaster@pikedom.com>
Date:   Sat Apr 28 20:09:29 2018 +0100

    Added .bash_profile

commit 08acbe7e6d7b483e9dc0a4b748dada50f195cc09
Author: Andrew Pike <hostmaster@pikedom.com>
Date:   Sat Apr 28 19:22:36 2018 +0100

    My initial bash config

Or git diff.

$ vcsh config-bash diff
diff --git a/.bashrc b/.bashrc
index 7b8ab85..e8252f1 100644
--- a/.bashrc
+++ b/.bashrc
@@ -96,6 +96,7 @@ alias free='free -m'                      # show sizes in MB
 alias np='nano -w PKGBUILD'
 alias more=less
 alias ll='ls -la'
+alias ls='ls -l'
 xhost +local:root > /dev/null 2>&1


vcsh works great but what happens when you want to migrate to or pull in the configuration to another machine? Enter mr and myrepos.


A command called mr, included in the myrepos package is used to simultaneously managed repositories. You can also use it to quickly enable or disable any repositories too. As mr knows about all the repositories you are interested in, you can save your mr configuration to Git and then proceed to pull in all your other configurations from their repositories in one go.


On Manjaro this would be:

$ yaourt -Sy myrepos

Some useful dependencies to consider when installing myrepos. Note you can use myrepos to manage repositories using many different revision control systems including Subversion and CVS!

Optional dependencies for myrepos
    bzr: support for bzr repositories
    cvs: support for cvs repositories
    darcs: support for darcs repositories
    git-annex: support for git-annex clones
    gitk: support for visualizing git repository history
    git: support for git repositories [installed]
    mercurial: support for mercurial repositories
    perl-html-parser: support for webcheckout
    perl-libwww: support for webcheckout
    perl-uri: support for webcheckout heuristically guessing partial URLs [installed]
    repo: support for repo repositories
    svn: support for subversion repositories [installed]
    unison: support for unison as a vcs
    vcsh: support for vcsh

Configure Mr

You need to create the file structure yourself. I’m using the recommended layout.

[andy@home-pc ~]$ mkdir -pv .config/mr/{available.d,config.d}
mkdir: created directory '.config/mr'
mkdir: created directory '.config/mr/available.d'
mkdir: created directory '.config/mr/config.d

And create the configuration file.

$ vim ~/.mrconfig

The content should look like the below.

git_gc = git gc "$@"
jobs = 5

include = cat ~/.config/mr/config.d/*

Add Repositories to mr

Now we need to add our repositories to mr. To add i3, create a config file in the available.d directory>

$ vim ~/.config/mr/available.d/i3.vcsh

Create a configuration file like so.

checkout = vcsh clone https://github.com/crazzyfool/config-i3.git config-i3
update = vcsh config-i3 pull 
push = vcsh config-i3 push
status = vcsh config-i3 status
gc = vcsh config-i3 gc

The first line is the path to the Git directory. Now lets add one for bash.

$ vim ~/.config/mr/available.d/bash.vcsh

And this one looks very similar.

checkout = vcsh clone https://github.com/crazzyfool/config-bash.git config-bash
update = vcsh config-bash pull
push = vcsh config-bash push
status = vcsh config-bash status
gc = vcsh config-bash gc

These repositories are now available to us but they are not enabled yet. To do so we need to create a symlink.

[andy@home-pc ~]$ cd ~/.config/mr/config.d/
[andy@home-pc config.d]$ ls -l
total 0
[andy@home-pc config.d]$ ln -s ../available.d/i3.vcsh . && ln -s ../available.d/bash.vcsh .
[andy@home-pc config.d]$ 
[andy@home-pc config.d]$ ls -l
total 0
lrwxrwxrwx 1 andy andy 24 Apr 29 18:58 bash.vcsh -> ../available.d/bash.vcsh
lrwxrwxrwx 1 andy andy 22 Apr 29 18:58 i3.vcsh -> ../available.d/i3.vcsh

Add a non-vcsh Repository

Now you can use vcsh to update and manage other repositories you use. For example, to configure Vim, I use The Ultimate Vim configuration.


To update you need to do a rebase:

cd ~/.vim_runtime
git pull --rebase

Lets try to accomplish this using mr…

$ cp -v ~/.config/mr/available.d/{bash,vim}.vcsh
$ vim ~/.config/mr/available.d/vim.vcsh

My one looks like the following.

checkout =
    git clone --depth=1 https://github.com/amix/vimrc.git --separate-git-dir ~/.config/vcsh/repo.d/config-vim.git ~/.vim_runtime
    sh ~/.vim_runtime/install_awesome_vimrc.sh

update =
    cd ~/.vim_runtime
    git pull --rebase

Don’t forget to enable it!

[andy@home-pc ~]$ cd ~/.config/mr/config.d/
[andy@home-pc config.d]$ ln -s ../available.d/vim.vcsh .
[andy@home-pc config.d]$ ls -la vim.vcsh 
lrwxrwxrwx 1 andy andy 23 Apr 29 22:54 vim.vcsh -> ../available.d/vim.vcsh

Now when you do a mr update for the first time, mr should automatically checkout and install the Ultimate Vim configuration.

$ mr update
mr update: /home/andy/.config/vcsh/repo.d/config-i3.git
Already up to date.

mr update: /home/andy/.config/vcsh/repo.d/config-bash.git
Already up to date.

mr checkout: /home/andy/.config/vcsh/repo.d/config-vim.git
Cloning into '/home/andy/.vim_runtime'...
Installed the Ultimate Vim configuration successfully! Enjoy 🙂

mr update: finished (3 ok)

If already checked-out and installed, it should check for updates and do a rebase if any are found.

[andy@home-pc ~]$ mr update
mr update: /home/andy/.config/vcsh/repo.d/config-i3.git
Already up to date.

mr update: /home/andy/.config/vcsh/repo.d/config-bash.git
Already up to date.

mr update: /home/andy/.config/vcsh/repo.d/config-vim.git
Already up to date.
Current branch master is up to date.

mr update: finished (3 ok)

Add any other Repositories

You can use mr to manage any other repositories you administer.

$ vim ~/.config/mr/available.d/scripts.vcsh

My one looks like:

checkout = 
    git clone https://github.com/crazzyfool/scripts.git ~/Scripts

update = 
    cd ~/Scripts
    git pull

push = 
    cd ~/Scripts
    git push

status = 
    cd ~/Scripts
    git status

Enable it.

$ cd ~/.config/mr/config.d/
$ ln -s ../available.d/scripts.vcsh .
$ cd
$ mr update

Add Mr to Git

For reasons that will become clear in the next and final section, it is a good idea to add your mr configuration to Github.

$ vcsh init config-mr
$ vcsh config-mr add .config/mr ~/.mrconfig
$ vcsh config-mr commit -m "Added initial mr commit"
vcsh config-mr remote add origin https://github.com/crazzyfool/config-mr.git
vcsh config-mr push -u origin master

Create mr configuration.

$ vim ~/.config/mr/available.d/mr.vcsh

Such as.

checkout = vcsh clone https://github.com/crazzyfool/config-mr.git config-mr
update = vcsh config-mr pull 
push = vcsh config-mr push
status = vcsh config-mr status
gc = vcsh config-mr gc

Enable it.

$ cd ~/.config/mr/config.d/
$ ln -s ../available.d/mr.vcsh .
$ cd

You’ll also need to create a gitignores file. My one looks like this.

$ cat ~/.gitignore.d/config-mr 

Add those files to the repository.

vcsh config-mr add ~/.config/mr/available.d/mr.vcsh ~/.config/mr/config.d/mr.vcsh ~/.gitignore.d/config-mr
vcsh config-mr commit -m "Added more files"
$ vcsh config-mr push

Pull Configuration to a New Host

This is ultimately why the vcsh/mr combination is so useful. Assuming you have installed vcsh and mr, getting setup should just be a simple case of.

$ vcsh clone https://github.com/crazzyfool/config-mr.git
$ mr update

However, as I’m not actually on a new host, I need to delete a load of files first.

$ rm -Rvf ~/.config/vcsh/repo.d/config-* ~/.config/mr/ ~/.mrconfig ~/.gitignore.d/ ~/.vim_runtime/ ~/.vimrc ~/Scripts/ ~/.bashrc ~/.bash_profile ~/.i3/config

The end result should be that all five repositories are pulled it.

$ vcsh clone https://github.com/crazzyfool/config-mr.git
Initialized empty shared Git repository in /home/andy/.config/vcsh/repo.d/config-mr.git/
Switched to a new branch 'master'
remote: Counting objects: 31, done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 31 (delta 4), reused 31 (delta 4), pack-reused 0
Unpacking objects: 100% (31/31), done.
From https://github.com/crazzyfool/config-mr
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> origin/master
[andy@home-pc ~]$ 
[andy@home-pc ~]$ vcsh list
[andy@home-pc ~]$ mr update
mr update: /home/andy/.config/vcsh/repo.d/config-mr.git
Already up to date.

mr checkout: /home/andy/Scripts
Cloning into '/home/andy/Scripts'...

mr checkout: /home/andy/.config/vcsh/repo.d/config-bash.git
Initialized empty shared Git repository in /home/andy/.config/vcsh/repo.d/config-bash.git/
Switched to a new branch 'master'
From https://github.com/crazzyfool/config-bash
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> origin/master

mr checkout: /home/andy/.config/vcsh/repo.d/config-i3.git
Initialized empty shared Git repository in /home/andy/.config/vcsh/repo.d/config-i3.git/
Switched to a new branch 'master'
From https://github.com/crazzyfool/config-i3
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> origin/master

mr checkout: /home/andy/.vim_runtime
Cloning into '/home/andy/.vim_runtime'...
Installed the Ultimate Vim configuration successfully! Enjoy 🙂

mr update: finished (5 ok)


As demonstrated, the combination of both vcsh and myrepos not only gives you a powerful way to manage and track the .conf files in your $HOME directory but also a very convenient way of managing all repositories you may use. It also means that reinstalling your OS or working on a new machine is not so much of a pain. 🙂

The Ultimate vim Configuration On Manjaro i3

vim is an awesome text editor (and more) but a good configuration can be a bit of a pain and personally I don’t have enough time to dedicate to it. The default vim configuration with Manjaro i3 is actually pretty good but I really like The Ultimate Vim Configuration.


Installation is simply on any Linux-based OS where Git is already present.

Install for a List of Users

The below installs it for the users andy and root.

$ git clone --depth=1 https://github.com/amix/vimrc.git ~/.vim_runtime
$ git clone --depth=1 https://github.com/amix/vimrc.git /opt/vim_runtime
$ sudo sh ~/.vim_runtime/install_awesome_parameterized.sh /opt/vim_runtime andy root

You must make sure all users have read access to/opt/vim_runtime when you are installing it for multiple users. Your user will need write access in order to be able to clone it.

Install for the Current User Only

To install if for the current user.

$ git clone --depth=1 https://github.com/amix/vimrc.git ~/.vim_runtime
$ sh ~/.vim_runtime/install_awesome_vimrc.sh

Install for a all Users

And for all users.

$ git clone --depth=1 https://github.com/amix/vimrc.git /opt/vim_runtime
$ sh ~/.vim_runtime/install_awesome_parameterized.sh /opt/vim_runtime --all

Don’t forget to make sure all users have read access to /opt/vim_runtime.

Give it a go – see what you think!


The Ultimate vim configuration uses the following fonts:

You can install these on Manjaro with.

$ yaourt -Sya ttf-hack adobe-source-code-pro-fonts ibm-plex-fonts


To update you can simply do the following.

$ cd ~/.vim_runtime
$ git pull --rebase


To uninstall, simply delete the ~/.vim_runtime directory plus any reference to it in ~/.vimr. For me this is the whole file.

$ rm -vRf ~/.vim_runtime
$ mv -v ~/.vimrc{,.bkup}

Clam Anti Virus on Arch Linux

Install Clam with:

sudo pacman -Sy clamav

To update the virus definitions, run:

sudo freshclam

You will also need to start and enable the clamam-freshclam.service to get the latest definitions at boot.

sudo systemctl enable --now clamav-freshclam.service

Make sure you have already run sudo freshclam before starting and enabling the clamav-daemon. If not, stop it and run.

sudo systemctl enable --now clamav-daemon.service

It is recommended to add the additional signatures from the AUR package, clamav-unofficial-sigs:

yaourt -Sy clamav-unofficial-sigs

Make sure the below is uncommented:

[andy@work-pc ~]$ grep user_configuration_complete /etc/clamav-unofficial-sigs/user.conf

Then to install and enable these unofficial signatures:

sudo clamav-unofficial-sigs.sh --install-all

To update the unofficial signatures, run:

sudo clamav-unofficial-sigs.sh

Lastly, scan for viruses:

mkdir ~/clamav-scan-results/
clamscan --recursive --infected --exclude-dir='^/sys|^/dev|^/proc|^/var/lib/clamav' --max-filesize=4000M --max-scansize=4000M / -l ~/clamav-scan-results/201803261436

Apache not Logging Correct IP when behind Incapsula WAF

Incapsula is a great resource to help protect your web site from unwanted traffic and attacks. It is a cloud-based application delivery platform, providing among other things:

  • Content Delivery Network (CDN)
  • Distributed Denial of Service (DDoS) Mitigation
  • Web Application Firewall (WAF)

Incapsula acts as a proxy, sitting in front of the nodes its protecting. The DNS points to Incapsula which hides the IP address to your site.  Incapsula analyses the traffic and removes any unwanted requests before passing it on to the web node.

As with any proxy-based system, the proxy rewrites the the X-Forwarded-For header information with the originating IP address.  However, Apache needs to be configured to use the header information.

Enable X-Forwarded-For

To enable X-Forwarded-For, open the main Apache configuration file and find the section that defines the LogFormat:

LogFormat "%v:%p %h %l %u %t \"%r\" %&gt;s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %&gt;s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %&gt;s %O" common
LogFormat "%{Referer}i -&gt; %U" referer
LogFormat "%{User-agent}i" agent

Then add the following additional line:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy

Lastly edit the configuration file for your virtual host:

# vim /etc/apache2/sites-enabled/pikedom.com.conf

Then comment out the existing CustomLog, combined in my example:

#CustomLog /var/www/pikedom.com/pikedom-access.log combined

And add a new entry for the CustomLog we created, proxy:

CustomLog /var/www/pikedom.com/pikedom-access.log proxy

Check Apache configuration for errors:

# apachectl -t

If none, restart Apache:

# service apache2 restart

To confirm X-Forward-For is working, first confirm what your public IP address is:

[andy@home-pc ~]$ curl -4 icanhazip.com

Then tail the access log and grep for your IP while visiting the site:

root@webhost1:~# tailf /var/www/pikedom.com/pikedom.com-access.log | grep - - [26/Mar/2018:10:39:02 +0100] "GET / HTTP/1.1" 301 325 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0" - - [26/Mar/2018:10:39:02 +0100] "GET / HTTP/1.1" 200 17576 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0" - - [26/Mar/2018:10:39:03 +0100] "GET /skin/frontend/pikedom/default/favicon.ico HTTP/1.1" 200 1243 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"

Job done!

Install Ansible on Arch Linux and Run Playbooks Locally

Installation is pretty simple….

$ sudo pacman -Sy ansible sshpass

Useful dependencies:

Optional dependencies for ansible
    sshpass: for ssh connections with password
    python2-passlib: crypt values for vars_prompt
    python2-netaddr: for the ipaddr filter
    python2-systemd: log to journal
    python2-pywinrm: connect to Windows machines
    python2-dnspython: for dig lookup
    python2-ovirt-engine-sdk: ovirt support
    python2-boto: aws_s3 module
    python2-jmespath: json_query support

We need sshpass for password-based authentication using Ansible.

Run Ansible Locally

Create an inventory hosts file:

$ sudo vim /etc/ansible/hosts

My one currently has just localhost as the control machine:


You’ll also need to make sure the SSH daemon process is running:

$ sudo systemctl start sshd
$ sudo systemctl enable sshd

Also, because by default SSH uses Host key checking which is not supported by sshpass, we first need to SSH locally so we add our fingerprint to our known_hosts file.

[andy@home-pc ~]$ ssh andy@
The authenticity of host ' (' can't be established.
ECDSA key fingerprint is SHA256:cynRa21eyQcfrYWjn4ajYiT1E7HBVn7mjfnDDsveiUE.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '' (ECDSA) to the list of known hosts.
andy@'s password: 
[andy@home-pc ~]$ exit
Connection to closed.
[andy@home-pc ~]$
[andy@home-pc ~]$ cat .ssh/known_hosts ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPu9ZWP54Dd5gkCllxGqlYyiwO+ZfHTmCLEJW0K9NR7ThLMxu3yTiQgzjWijN80sEs14U4GHNoc8Qv4DujaVmzU=

Now we can test it works by using the ansible ping module:

[andy@home-pc ~]$ ansible control -m ping -u andy --ask-pass
SSH password: | SUCCESS => {
    "changed": false, 
    "ping": "pong"

Now finally you can run ansible commands locally on your machine.

[andy@home-pc ~]$ ansible control -u andy --ask-pass -a "free -h"
SSH password: | SUCCESS | rc=0 >>
              total        used        free      shared  buff/cache   available
Mem:           15Gi       732Mi        14Gi        48Mi       638Mi        14Gi
Swap:          34Gi          0B        34Gi

XWiki on Ubuntu 16.04 LTS with Nginx Reverse Proxy

Install XWiki and all dependant programs on a 4 GB cloud server. This means:

1) Java
2) Tomcat
3) MySQL/MariaDB
4) XWiki
5) Nginx

Here we use Nginx as a reverse proxy to firstly redirect all HTTP to HTTPS and then forward all requests on port 80/443 to port 8080 (tomcat) on the localhost. Here I use LetsEncrytp for my SSL certificates.

Before we begin…

Setup DNS

wiki.dummydomains.org.uk ——>

Prepare the Server


Update and reboot the server.

apt-get update
apt-get dist-upgrade

Enable the firewall

ufw status
ufw enable
ufw allow ssh
ufw reload
ufw status

Install Oracle Java

This is a requirement before installing Tomcat or XWiki. At the time of writing, Java 8.x is recommended as 9.x is too new and has a number of known bugs still.

apt-get install software-properties-common
add-apt-repository ppa:webupd8team/java
apt-get update
apt-get install oracle-java8-installer

You will need to accept the license agreement:

Accept license agreement

Binary code license terms

Because many programs check for $JAVA_HOME, it is a good idea to set it now. If you don’t know the path, check with:

root@wiki:~# update-alternatives --config java
There is 1 choice for the alternative java (providing /usr/bin/java).

  Selection    Path                                     Priority   Status
  0            /usr/lib/jvm/java-8-oracle/jre/bin/java   1081      auto mode
* 1            /usr/lib/jvm/java-8-oracle/jre/bin/java   1081      manual mode

Press &amp;lt;enter&amp;gt; to keep the current choice[*], or type selection number:

Then edit your system $PATH variable so that the /usr/lib/jvm/java-8-oracle is the first path.

nano /etc/environment

Mine looks like this:

root@wiki:~# cat /etc/environment

You will need to log out and back in first but you can test with the below command.

root@wiki:~# echo $JAVA_HOME

Create Virtual Host and Generate SSL

Install Nginx and LetsEncrypt.


Install and configure Nginx.

apt-get install apache2-utils nginx
systemctl enable nginx

Create a very basic virtual host by editing the nginx configuration file and inserting your server name in the server_name variable.

vim /etc/nginx/sites-enabled/default

Mine looks like this.

root@wiki:~# egrep -v "^$|^[[:space:]]*#" /etc/nginx/sites-available/default 
server {
	listen 80 default_server;
	listen [::]:80 default_server;
	root /var/www/html;
	index index.html index.htm index.nginx-debian.html;
	server_name wiki.dummydomains.org.uk;
	location / {
		try_files $uri $uri/ =404;


systemctl restart nginx

Check it works!

Nginx test page
If it doesn’t, check the firewall…

Allow HTTP and HTTPS Traffic

If you use a local firewall like UFW or iptables, you will need to allow port 80 and 443.

ufw status
ufw allow http
ufw allow https
ufw status
ufw reload


add-apt-repository ppa:certbot/certbot
apt-get update
apt-get install python-certbot-nginx
certbot --nginx -d wiki.dummydomains.org.uk -d dummydomains.org.uk

Your certificate will get saved to /etc/letsencrypt/live/wiki.dummydomains.org.uk.

LetsEncrypt will edit your virtual hosts file.  The parts we are interested in are:

  listen 443 ssl; # managed by Certbot
  ssl_certificate /etc/letsencrypt/live/wiki.dummydomains.org.uk/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/letsencrypt/live/wiki.dummydomains.org.uk/privkey.pem; # managed by Certbot
  include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

  if ($scheme != "https") {
    return 301 https://$host$request_uri;
  } # managed by Certbot


Install XWiki

Install XWiki.

wget -q "https://maven.xwiki.org/public.gpg" -O- | apt-key add -
wget "https://maven.xwiki.org/stable/xwiki-stable.list" -P /etc/apt/sources.list.d/
apt-get update

Search for XWiki packages to install.

apt-cache search xwiki

According to the official documentation, the enterprise version is out-of-date and the non-enterprise version should be used.

apt-get install xwiki-tomcat8-mysql

Set the root MySQL password:


Set root MySQL password
Set root MySQL password

When asked if you should configure the database with dbconfig-common, say yes.

Configure with dbconfig-common
Configure with dbconfig-common

MySQL application password:


Select application password
Select application password

Check tomcat8 is listening on port 8080:

root@wiki:~# netstat -plnt | grep :8080
tcp6       0      0 :::8080                 :::*                    LISTEN      15840/java

Check your memory usage:

root@wiki:~# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.9G        487M        1.9G         10M        1.5G        3.3G
Swap:            0B          0B          0B

You will need to increase the default about of memory allocated to Java. Here’s how:

vim /etc/default/tomcat8


root@wiki:~# grep ^JAVA_OPTS /etc/default/tomcat8
JAVA_OPTS="-Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC"


root@wiki:~# grep ^JAVA_OPTS /etc/default/tomcat8
JAVA_OPTS="-Djava.awt.headless=true -Xmx1024m -XX:+UseConcMarkSweepGC"

Restart Tomcat

systemctl restart tomcat8

Test using using a browser:


If you’ve enabled a firewall and you want to test:

ufw allow 8080/tcp
ufw reload

However I’m not going to do this – I’m going to setup Nginx as a proxy first.

Configure Nginx

Remove the default virtual host configuration.

rm -v /etc/nginx/sites-enabled/default.conf
vim /etc/nginx/sites-available/wiki.dummydomains.org.uk.conf

My site configuration look as follows:

upstream tomcat {
  server fail_timeout=0;
  keepalive 64;

server {
  listen [::]:80;
  listen ssl;
  listen [::]:443 ssl;
  server_name wiki.dummydomains.org.uk dummydomains.org.uk;
  ssl_certificate /etc/letsencrypt/live/wiki.dummydomains.org.uk/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/wiki.dummydomains.org.uk/privkey.pem;
  include /etc/letsencrypt/options-ssl-nginx.conf;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
  # redirect http to https
  if ($scheme != "https") {
    return 301 https://$host$request_uri;

  auth_basic "Authentication Required";
  auth_basic_user_file xwiki-access;

  location / {
    client_max_body_size 20M;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass_request_headers on;
    proxy_set_header Connection "keep-alive";
    proxy_store off;
    proxy_headers_hash_max_size 512;
    deny all;

    proxy_pass http://tomcat/;

I also want to password protect my wiki:

htpasswd -c /etc/nginx/xwiki-access andy

Enable the site:

cd /etc/nginx/sites-enabled/
ln -s ../sites-available/xwiki.dummydomains.org.uk.conf .

….and check configuration file for errors.

root@wiki:~# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Check config and restart Nginx:

systemctl restart nginx

Now try entering the following into your browser and complete the on-screen installation instructions:


Complete Installation

Here are few things I normally do after an installation.

Make Root Application

I want to make this Wiki instance the root web application and remove the trailing /xwiki from the URL.

systemctl stop tomcat8.service
mv -v /etc/tomcat8/Catalina/localhost/xwiki.xml /etc/tomcat8/Catalina/localhost/ROOT.xml
vim /etc/xwiki/xwiki-tomcat8.xml


<Context path="/xwiki" docBase="/usr/lib/xwiki" privileged="true" crossContext="true">
  <!-- make symlinks work in Tomcat -->
  <Resources allowLinking="true" />


<Context path="/" docBase="/usr/lib/xwiki" privileged="true" crossContext="true">
  <!-- make symlinks work in Tomcat -->
  <Resources allowLinking="true" />

Don’t forget to start Tomcat again:

systemctl start tomcat8.service

Now the URL is simply:


Enable superadmin

This is needed if you plan to import XWiki pages from a previous installation.

vim /etc/xwiki/xwiki.cfg

Find the following section.

#-# Enable to allow superadmin. It is disabled by default as this could be a
#-# security breach if it were set and you forgot about it. Should only be enabled
#-# for recovering the Wiki when the rights are completely messed.
# xwiki.superadminpassword=system

….and change to:

#-# Enable to allow superadmin. It is disabled by default as this could be a
#-# security breach if it were set and you forgot about it. Should only be enabled
#-# for recovering the Wiki when the rights are completely messed.

Don’t forget to restart Tomcat if necessary.

Update Cookie Encryption Keys

When a user logs in, three cookies are saved to their machine. These cookies are encrypted with the below details. First we need to get the two random strings of equal length.

root@wiki:~# date +%s | sha256sum | base64 | head -c 32 ; echo
root@wiki:~# date +%s | sha256sum | base64 | head -c 32 ; echo

Then edit the xwiki.cfg file.

vim /etc/xwiki/xwiki.cfg

Find the relevant section and edit to look like the below.


Don’t forget to restart Tomcat if necessary.

Complete the Installation

Login to complete the installation.

Log in
Log in

Click continue.

Installation wizard
Installation wizard

Register and log in.

Register and login
Register and login


Install xwiki
Install xwiki

Select 9.9 and continue.

Install 9.9
Install 9.9

Confirm installation again.

Confirm installation
Confirm installation



Continue again.

Continue again
Continue again

Confirm the report by clicking continue.

Confirm installation report
Confirm installation report

Installation complete!

Installation Complete
Installation Complete

Import old XWiki Content

Lets see if the import feature works!  Log in as the superadmin user and then navigate to the Administration section:


Then select Content, followed by Import:


Select the backup.xar that you (hopefully) took earlier and import all the content.

Package content
Package content

Select the following options.

Import options
Import options


Installing tox on Manjaro i3

Tox is an open source secure alternative to the likes of Skype. To install it, you need to install the core package and one of the available GUIs.  You can compare some of the available clients here but personally I like qtox. The below will pull in the required dependencies.

sudo pacman -Sy qtox

If you want the latest Git version, you can install qtox-git from the AUR.

yaourt -S qtox-git

Make sure you run that as a regular user – not root.  This pulls in the core package from the ABS repository also.

To add a contact, you will need their Tox ID – which looks like this:


If that is a bit too annoying for you, you can use a free ToxDNS provider, such as utox.org. This will give you a “username@utox.org” address to give your people.