Migrate GitLab Instance to new Host

Here I migrate a GitLab installation from one server to another. The hostname and IP addresses will be different. Both were installed from the Omnibus version. However, currently the exact version numbers differ.

In order to migrate successfully, both versions must be exactly the same type and version. For example, both installed using the Omnibus version and exactly the same version number.

If you are unsure whether you installed the Omnibus version, you can confirm with the below command. This will fail if not. If it returns output similar to the below, you have the Omnibus version. It also shows me that currently have version 10.2.3 installed.

root@repo:~# gitlab-rake gitlab:env:info

System information
System:         Ubuntu 16.04
Current User:   git
Using RVM:      no
Ruby Version:   2.3.5p376
Gem Version:    2.6.13
Bundler Version:1.13.7
Rake Version:   12.1.0
Redis Version:  3.2.5
Git Version:    2.13.6
Sidekiq Version:5.0.4
Go Version:     unknown

GitLab information
Version:        10.2.3
Revision:       3141105
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://gitlab.openit-uk.com
HTTP Clone URL: https://gitlab.openit-uk.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.openit-uk.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: no

GitLab Shell
Version:        5.9.4
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks
Git:            /opt/gitlab/embedded/bin/git

If you do not have the Omnibus version, consult the official documentation on how to migrate. You need to make sure the version you are going to and from are the same.

The Strategy

  1. Install GitLab on new server [10.7.3]
  2. Update old GitLab to the latest version [10.7.3]
  3. Backup old GitLab
  4. Transfer backup files to new host
  5. Restore from backup on new instance

Install GitLab

Install GitLab on the new server.

Update GitLab

This can simply be achieved by updating the system.

apt-get update
apt-get dist-upgrade

GitLab should now be updated to the latest version. It should also have created an application backup in the below location as part of the process.

root@repo:~# ls -l /var/opt/gitlab/backups/
total 204
-rw------- 1 git git  71680 May 16 19:29 1526498956_2018_05_16_10.2.3_gitlab_backup.tar

Check the version is now the latest and the same as the origin server.

root@repo:~# gitlab-rake gitlab:env:info | grep "GitLab information" -A1
GitLab information
Version:	10.7.3

Backup GitLab

To fully backup, you need to take both a configuration and a application backup. The configuration backup includes all of GitLabs configuration including the external URL. The application backup includes all of your repositories.

Configuration Backup

All GitLab configuration is inside /etc/gitlab. To backup this directory, run:

sudo sh -c 'umask 0077; tar -cf $(date "+etc-gitlab-%s.tar") -C / etc/gitlab'

This creates the following file in the current working directory.

root@repo:~# ls -la etc-gitlab-1526500078.tar
-rw------- 1 root root 102400 May 16 19:47 etc-gitlab-1526500078.tar

With the following content.

root@repo:~# tar -tvf etc-gitlab-1526500078.tar
drwxrwxr-x root/root         0 2017-12-09 23:32 etc/gitlab/
drwxr-xr-x root/root         0 2017-12-06 14:58 etc/gitlab/trusted-certs/
drwx------ root/root         0 2017-12-09 23:41 etc/gitlab/ssl/
-rw-r--r-- root/root      2025 2017-12-09 23:09 etc/gitlab/ssl/repo.pikedom.com.crt
-rw-r--r-- root/root      1708 2017-12-09 23:08 etc/gitlab/ssl/repo.pikedom.com.key
-rw-r--r-- root/root      2025 2017-12-09 23:39 etc/gitlab/ssl/gitlab.openit-uk.com.crt
-rw-r--r-- root/root      1708 2017-12-09 23:41 etc/gitlab/ssl/gitlab.openit-uk.com.key
-rw------- root/root     14906 2018-05-16 19:31 etc/gitlab/gitlab-secrets.json
-rw------- root/root     72812 2017-12-09 23:32 etc/gitlab/gitlab.rb

Application Backup

By default, backups are kept in /var/opt/gitlab/backups.

root@repo:~# grep backup_path /etc/gitlab/gitlab.rb
# gitlab_rails['manage_backup_path'] = true
# gitlab_rails['backup_path'] = "/var/opt/gitlab/backups"

If you change this location, be sure to run the below after.

sudo gitlab-ctl reconfigure

To actually take the application backup, run the following.

sudo gitlab-rake gitlab:backup:create

This creates a backup in the following location.

root@repo:~# ls -ltr /var/opt/gitlab/backups/
total 204
-rw------- 1 git git  71680 May 16 19:29 1526498956_2018_05_16_10.2.3_gitlab_backup.tar
-rw------- 1 git git 133120 May 16 20:14 1526501659_2018_05_16_10.7.3_gitlab_backup.tar

Host Backup (optional)

If rebuilding the machine and keeping the same IP, to avoid having to delete the host key entry in the ~/.ssh/know_hosts file, run the following to backup the SSH host keys.

sudo tar -cvf $(date "+hostkeys-%s.tar") $(grep HostKey /etc/ssh/sshd_config | grep -v ^'#' | awk '{print $2}')

Transfer

Lets zip and compress these files into a single archive for transfer. Although I don’t plan on restoring the host keys, here I zip and compress all three all the same.

/var/opt/gitlab/backups/1526501659_2018_05_16_10.7.3_gitlab_backup.tar
/root/etc-gitlab-1526500078.tar
/root/hostkeys-1526501148.tar

Zip and compress with the following.

sudo tar -cvzf ~/gitlab-backup.tar.gz /var/opt/gitlab/backups/1526501659_2018_05_16_10.7.3_gitlab_backup.tar /root/etc-gitlab-1526500078.tar /root/hostkeys-1526501148.tar

Use scp to copy the backup securely over the internet. Here I run the below from the new GitLab server to pull the above backup to the current working directory.

sudo scp -v root@old.dummydomains.org.uk:/root/gitlab-backup.tar.gz .

Restore

As I am mainly concerned about the Git repositories themselves, I will only be doing an application restore.

If you are moving to a new host with the same hostname etc., you should also restore the configuration backup to /etc/gitlab/. If you want everything to be the same but the hostname is different, you can manually amend anything such as external_url before running sudo gitlab-ctl reconfigure.

Application Backup Restore

Stop processes using the database.

sudo gitlab-ctl stop unicorn
sudo gitlab-ctl stop sidekiq
sudo gitlab-ctl status

Check they’re actually down.

sudo gitlab-ctl status | grep ^down

Move the application backup to the backups directory as defined in gitlab.rb and correct the owner if necessary.

sudo mv -v ~/temp/1526501659_2018_05_16_10.7.3_gitlab_backup.tar /var/opt/gitlab/backups/
sudo chown -v git:git /var/opt/gitlab/backups/1526501659_2018_05_16_10.7.3_gitlab_backup.tar

Then restore.

sudo gitlab-rake gitlab:backup:restore BACKUP=1526501659_2018_05_16_10.7.3

Now restart GitLab.

sudo gitlab-ctl start

And check for problems.

sudo gitlab-rake gitlab:check SANITIZE=true

If all was successful, you should be able to log into the new site as before with whatever users you had setup. If you had 2FA enabled, you may need to restore the gitlab-secrets.json.

Configuration Backup Restore

Exactly what you choose to restore from the /etc/gitlab/ directory is up to you. None maybe necessary. If any of your users had 2FA setup, you will need to restore the gitlab-secrects.json file or else they will lose their access. If unsure, you could always do a diff to compare the differences.

diff old-gitlab.rb new-gitlab.rb

If moving to a new host but the hostname and everything should remain the same, you can simply move the old directory out the way and replace with the old.

First, stop unicorn and sidekiq at a minimum.

sudo gitlab-ctl stop

Then restore the configuration directory.

mv -v /etc/gitlab{,.original}
mv -v ~/temp/etc/gitlab /etc/gitlab

Then start GitLab again and check.

sudo gitlab-ctl start
sudo gitlab-rake gitlab:check SANITIZE=true

Fix any issue and then re-run if necessary. Try re-configuring if you have issues.

 
sudo gitlab-ctl reconfigure

If even the IP address remains the same as before and you want to prevent your users from seeing an SSH man-in-the-middle attack warning, you will also need to restore the HostKeys.

You need to replace the HostKeys defined in your sshd_config.

andy@repo:~$ grep HostKey /etc/ssh/sshd_config 
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key

…with the ones from your backup.

andy@repo:~$ tar -tvf temp/hostkeys-1526501148.tar.gz 
-rw------- root/root      1679 2017-12-05 13:37 etc/ssh/ssh_host_rsa_key
-rw------- root/root       672 2017-12-05 13:37 etc/ssh/ssh_host_dsa_key
-rw------- root/root       227 2017-12-05 13:37 etc/ssh/ssh_host_ecdsa_key
-rw------- root/root       399 2017-12-05 13:37 etc/ssh/ssh_host_ed25519_key

Rename the existing keys.

for i in $(grep ^HostKey /etc/ssh/sshd_config | awk '{print $2}'); do sudo mv -v $i $i.bkp-$(date +%s);done

Then restore the old ones in-place.

sudo tar -xvzf temp/hostkeys-1526501148.tar.gz -C /

Restart GitLab and check for errors.

sudo gitlab-ctl restart
sudo gitlab-rake gitlab:check SANITIZE=true

Resources

https://www.fabian-keller.de/blog/migrating-a-gitlab-instance-to-another-server
https://www.tienle.com/2016/05-24/migrate-gitlab-to-new-server.html
https://stackoverflow.com/questions/31534293/how-to-migrate-gitlab-to-a-new-server

Install and Configure GitLab on Ubuntu 16.04

Dependencies

Run to the below command to install the GitLab dependencies.

sudo apt-get update
sudo apt-get install ca-certificates curl openssh-server postfix

When prompted, select Internet Site when configuring Postfix.

Postfix - Mail Type
Postfix – Mail Type

Enter the FQDN.

Postfix - enter FQDN
Postfix – enter FQDN

Installing GitLab

GitLab provides a bash script that add their repository for you. Just download it and run it.

cd /tmp
curl -LO https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh
sudo bash /tmp/script.deb.sh

Now you can install the GitLab package using the apt-get package manager.

sudo apt-get install gitlab-ce

Configuring GitLab

You need to first edit the configuration file.

sudo vim /etc/gitlab/gitlab.rb

I added the following line to define my primary hostname and URL.

external_url 'https://repo.dummydomains.org.uk'

I also enabled letsencrypt. If you do, make sure the DNS is setup already.

letsencrypt['enable'] = true
letsencrypt['contact_emails'] = ['postmaster@dummydomains.org.uk']
letsencrypt['auto_renew'] = true
letsencrypt['auto_renew_hour'] = "3"
letsencrypt['auto_renew_minute'] = "30"
letsencrypt['auto_renew_day_of_month'] = "*/7"

This will auto-renew at 3:30 AM every seventh day. Now run the below command to bring about your changes.

sudo gitlab-ctl reconfigure

If all was successful, you should see gitlab Reconfigured! on the last line of output.

Browse to the URL and change your password.

GitLab - Change Password
GitLab – Change Password

Then log in with the root user and the password you just created.

GitLab - Sign In
GitLab – Sign In

GitLab is now installed!

GitLab - Sign In
GitLab – Sign In

It’s time to start configuring it! You may want to look at preventing people signing up.

Resources

https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-gitlab-on-ubuntu-16-04
https://docs.gitlab.com/omnibus/
https://about.gitlab.com/installation/#ubuntu
https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https
https://docs.gitlab.com/omnibus/settings/ssl.html#let-39-s-encrypt-integration
https://www.howtoforge.com/tutorial/how-to-install-and-configure-gitlab-on-ubuntu-16-04/
https://tecadmin.net/install-gitlab-ce-on-ubuntu/
https://hostpresto.com/community/tutorials/install-and-configure-gitlab-on-ubuntu-16-04/

Apache not Logging Correct IP when behind Incapsula WAF

Incapsula is a great resource to help protect your web site from unwanted traffic and attacks. It is a cloud-based application delivery platform, providing among other things:

  • Content Delivery Network (CDN)
  • Distributed Denial of Service (DDoS) Mitigation
  • Web Application Firewall (WAF)

Incapsula acts as a proxy, sitting in front of the nodes its protecting. The DNS points to Incapsula which hides the IP address to your site.  Incapsula analyses the traffic and removes any unwanted requests before passing it on to the web node.

As with any proxy-based system, the proxy rewrites the the X-Forwarded-For header information with the originating IP address.  However, Apache needs to be configured to use the header information.

Enable X-Forwarded-For

To enable X-Forwarded-For, open the main Apache configuration file and find the section that defines the LogFormat:

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

Then add the following additional line:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy

Lastly edit the configuration file for your virtual host:

# vim /etc/apache2/sites-enabled/pikedom.com.conf

Then comment out the existing CustomLog, combined in my example:

#CustomLog /var/www/pikedom.com/pikedom-access.log combined

And add a new entry for the CustomLog we created, proxy:

CustomLog /var/www/pikedom.com/pikedom-access.log proxy

Check Apache configuration for errors:

# apachectl -t

If none, restart Apache:

# service apache2 restart

To confirm X-Forward-For is working, first confirm what your public IP address is:

[andy@home-pc ~]$ curl -4 icanhazip.com
180.112.113.2

Then tail the access log and grep for your IP while visiting the site:

root@webhost1:~# tailf /var/www/pikedom.com/pikedom.com-access.log | grep 180.112.113.2
180.112.113.2 - - [26/Mar/2018:10:39:02 +0100] "GET / HTTP/1.1" 301 325 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"
180.112.113.2 - - [26/Mar/2018:10:39:02 +0100] "GET / HTTP/1.1" 200 17576 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"
180.112.113.2 - - [26/Mar/2018:10:39:03 +0100] "GET /skin/frontend/pikedom/default/favicon.ico HTTP/1.1" 200 1243 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"
^C
root@webhost1:~#

Job done!

Installing Ansible on Ubuntu 14.04 LTS

Take a look at the official installation guide. The simplest way to install Ansible on Ubuntu is add the PPA repository and install via apt-get.

If not already installed, you will need the software-properties-common package.

sudo apt-get install software-properties-common

Then add the repository and install ansible.

sudo apt-add-repository ppa:ansible/ansible
sudo apt-get update
sudo apt-get install ansible

Presumably you’ve already got an external server that you want to configure with Ansible. You will need SSH access, and if you’ve not already done so, you’re gonna want to setup key-based authentication. Assuming you’ve done that, you can test things are working with:

su -
mv -v /etc/ansible/hosts{,.original}
echo 134.213.48.162 > /etc/ansible/hosts
exit

I also need to tell ansible to connect as the root user.

sudo mkdir /etc/ansible/group_vars
sudo vim /etc/ansible/group_vars/all

Enter the following. The three dashes at the top indicate this is a yaml file.

---
ansible_ssh_user: root

You should now be able to test with the following.

ansible -m ping all

You should see output similar to the below if all went well.

andy@bastion:~$ ansible -m ping all
134.213.48.162 | success >> {
"changed": false,
"ping": "pong"
}

X2Go on Ubuntu Server 14.04

In a previous post, I talked about my experience using X2Go with XFCE4 and Lubuntu.

XFCE4 via X2Go

Here is how it was achieved…..

On the Server

Here I’m using Ubuntu 14.04 LTS but you can install X2Go on just about any Linux distro.

Create a User Account

Create a regular user to run the desktop session under.

sudo useradd -m -s /bin/bash andrew
sudo passwd andrew

For your own sanity, I recommend you setup password-less key-based authentication as soon as possible……go, do it now!

Configure SSH

Open the main configuration file for the OpenSSH daemon process.

sudo vim /etc/ssh/sshd_config

Ensure X11 forwarding is enabled.


X11Forwarding yes

Don’t forget to test for configuration errors and restart the SSH service.

sudo sshd -t
sudo service ssh restart

Install Lightweight Desktop Environment

Both Lubuntu and XFCE4 work well, out-the-box with X2go. I installed both side-by-side for testing and both worked well together. You can even pause/suspend your Lubuntu or XFCE4 session and come back to it another time.

For XFCE4

sudo apt-get install xfce4

Note, for some reason you will also need to install the following packages or you will have missing icons.

sudo apt-get install gnome-icon-theme-full tango-icon-theme

For Lubuntu

sudo apt-get install lubuntu-desktop

Along with (a load of) other packages, you will now have XOrg installed. This means, as long as X11 forwarding has been enabled on the client side of the SSH connection, you can now test X11 with a program like firefox if you have it installed already.

Install X2Go Server Software

Install the repository package if it’s not already installed.

sudo apt-get install software-properties-common

Add the X2Go repository and install packages. If using Ubuntu 10.04 or 12.04, install python-software-properties instead of the software-properties-common package.

sudo add-apt-repository ppa:x2go/stable
sudo apt-get update
sudo apt-get install x2goserver x2goserver-xsession

For information about other distributions, see the X2Go server installation page.

DE Bindings

If you installed Lubuntu, you may want to install the following package for Desktop Environment bindings. I don’t believe there is currently a desktop bindings package for XFCE4.

sudo apt-get install x2golxdebindings

This is probably a good place to reboot if like me you’ve installed a lot of new packages.

On the Client

Again, you can install the client on just about anything – including Windows! Here I am using Manjaro i3 Community Linux.

sudo pacman -S x2goclient

X2Go also has some other clients that look useful – like a Python one for example.

SSH Client Configuration

Make sure you have at least ForwardX11 yes in /etc/ssh/ssh_config or ~/.ssh/config.


Host *
ForwardX11 yes
ForwardX11Trusted yes

Test X11 Forwarding

You can test that X11 is being forwarded correctly by using SSH to log onto the remove server, from your local Linux desktop, and and issue the following.

andrew@loader:~$ firefox &
[1] 2257

You might also want to setup password-less SSH key-based authentication if you’ve not done so already.

Create Sessions

For Lubuntu, select Custom Desktop and enter the below for the command.

lxsession -e LXDE -s Lubuntu

For XFCE4, you can just select XFCE.

Lubuntu XFCE4

I’m using the i3wm – I found the best result in appearance using the Use whole display option under the Input/Output tab, and then select the display (monitor) you want to use.

i3wm compatible

Resources

https://wiki.archlinux.org/index.php/X2Go
https://www.howtoforge.com/tutorial/x2go-server-ubuntu-14-04/
http://wiki.x2go.org/doku.php/doc:installation:x2goserver
https://bugs.launchpad.net/ubuntu/+source/lubuntu-default-settings/+bug/1241958
http://ubuntuforums.org/showthread.php?t=2228137

Minimal Desktop Environment over SSH

So I wanted to install a Java desktop application and have it publicly available on a server somewhere. Using a light weight desktop environment on one of my cloud servers made sense – provided that is, I could find something reasonably secure.

I came across X2Go and decided to give it a try on one of my Rackspace cloud servers. I used a 2 GB General Purpose v1 server and was surprised at how low the resource usage was – and consequentially how quick and responsive it all felt.

X2Go is a remote desktop tool that uses the NX technology protocol and operates entirely over a secure SSH connection. Using SSH keys makes the process of logging in pretty painless too!

I’m using Ubuntu 14.04 LTS for the OS, on the server and Manjaro i3 community edition on my local desktop, as the client. On the server I tried both XFCE4 and Lubuntu as the Desktop Environments.

Lubuntu via X2Go

Personally I think I prefer XFCE4 as it was slightly easier to install and lightning quick to use. When I used Lubuntu, the start menu could take a while (like a minute!) to load. Once it had loaded though, it too was lightning quick. And to be fair to Lubuntu, I didn’t really look into it much further.

XFCE4 using X2Go

As a note to my future self, here’s what you need to do.

Rackspace Cloud Monitoring Agent

The Rackspace cloud monitoring agent allows you to monitor CPU, memory, filesystem usage and system processes. It does this by collecting information about the system and pushing it out to Rackspace Cloud Monitoring web services, where they can be analyzed, graphed, and alerted on. It is this technology that the Rackspace monitoring checks are built upon.

Plus you get a nice pretty little bar graph in the server details section of the control panel 🙂

Rackspace monitoring agent

Install the Agent

While the instructions used here are for Ubuntu 14.04 LTS, this page lists the exact commands needed for all major distros.

wget http://meta.packages.cloudmonitoring.rackspace.com/ubuntu-14.04-x86_64/rackspace-cloud-monitoring-meta-stable_1.0_all.deb
dpkg -i rackspace-cloud-monitoring-meta-stable_1.0_all.deb
apt-get update
apt-get install rackspace-monitoring-agent

If your distribution of choice isn’t listed, you can always install from source.

Configure and Start Daemon

If the /etc/rackspace-monitoring-agent.cfg file isn’t present, you will need to choose one of the methods below to start the service.

Quick Method

Run the below commands, replacing the username and API key with your own.

rackspace-monitoring-agent --setup --username <your-username> --apikey <your-api-key>
rackspace-monitoring-agent start -D

Interactive Method

Alternatively you can simply run the below to interactively enter your username and your API key or password.

rackspace-monitoring-agent --setup

Followed by…

service rackspace-monitoring-agent start

Updating

The monitoring agent does not update itself. However, if you installed using a package manager, such as apt-get, agent updates will be pulled in and applied with regular system updates anyway.

apt-get update
apt-get dist-upgrade

Uninstalling the Agent

Assuming you didn’t install from source and you used your distros package manager, you will uninstall with the same method. I am using Ubuntu, so…

apt-get remove rackspace-monitoring-agent

Or if you’re using CentOS/RHEL.

yum remove rackspace-monitoring-agent

Related Documents

https://github.com/virgo-agent-toolkit/rackspace-monitoring-agent

http://www.rackspace.com/knowledge_center/article/install-and-configure-the-cloud-monitoring-agent#UpgradeAgent

http://meta.packages.cloudmonitoring.rackspace.com/

http://docs.rackspace.com/cm/api/v1.0/cm-devguide/content/install-configure.html

http://www.rackspace.com/knowledge_center/article/about-the-cloud-monitoring-agent

Installing Oracle Java on Ubuntu 14.04 LTS

You will need to run the following as the root user. To install Java 7…

add-apt-repository ppa:webupd8team/java
apt-get update
apt-get install oracle-java7-installer

…and for Java 8…

add-apt-repository ppa:webupd8team/java
apt-get update
apt-get install oracle-java8-installer

You will have to agree and accept the following licenses:

Oracle Java 8 - Accept License

Oracle Java 8 - Accept Binary License

You can quickly test both java and javac with the following….

java -version
javac -version

…or you may want to check out one of my previous posts here.

Related Documents

https://salayhin.wordpress.com/2014/04/23/install-oracle-java-78-jdk-and-jre-in-ubuntu-14-04/

https://www.digitalocean.com/community/tutorials/how-to-install-java-on-ubuntu-with-apt-get

http://askubuntu.com/questions/521145/how-to-install-oracle-java-on-ubuntu-14-04

Protect Your Cloud Infrastructure Servers with Isolated Cloud Networks

Create a Private Cloud Network

Create an isolated cloud network. Here I am using the supernova client to communicate with the Rackspace OpenStack API.

supernova uk network-create "Infrastructure" "192.168.3.0/24"
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| cidr     | 192.168.3.0/24                       |
| id       | 4d15b8ad-45c5-4169-a4fa-d36f1a776efd |
| label    | Infrastructure                       |
+----------+--------------------------------------+

Take note of the id – you’ll need it shortly!

Create a Proxy Server and Attach to the Private Network

supernova uk boot proxy-bast --flavor 2 --image 189678ca-fe2c-4b7a-a986-30c3660edfa5 --nic net-id=4d15b8ad-45c5-4169-a4fa-d36f1a776efd

The above creates a server using the CentOS 6.6 image. Other images of interest are:

+--------------------------------------+------------------------------------------+--------+
| ID                                   | Name                                     | Status |
+--------------------------------------+------------------------------------------+--------+
| 189678ca-fe2c-4b7a-a986-30c3660edfa5 | CentOS 6 (PVHVM)                         | ACTIVE |
| f8ae535e-67c0-41a5-bf55-b06d0ee40cc2 | CentOS 7 (PVHVM)                         | ACTIVE |
| 6909f56c-bd77-411a-8c0e-c37876b68d1d | Ubuntu 14.04 LTS (Trusty Tahr) (PVHVM)   | ACTIVE |
+--------------------------------------+------------------------------------------+--------+

Proxy Bastion Configuration

Later we create a cloud server with no public IP, which is protected by sitting behind our proxy bastion. From the bastion side, in order for our protected server to have access to the internet, we need to apply firewall rules for IP forwarding and Network Address Translation. This process differs depending on which distribution you use. Here I cover CentOS 6.6, CentOS 7 and Ubutnu 14.04.

CentOS 6.6

Under CentOS 6.6 and before, you need to configure IPTables to do the forwarding and the Network Address Translation (NAT). We will be forwarding the traffic from the eth2 interface, out through the eth0 interface. We also use Static NAT or MASQUERADE so that traffic coming from our protected infrastructure, takes on the public IP address of our proxy bastion.

2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether bc:76:4e:08:40:d8 brd ff:ff:ff:ff:ff:ff
    inet 95.138.163.75/24 brd 95.138.163.255 scope global eth0
    inet6 2a00:1a48:7805:113:be76:4eff:fe08:40d8/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::be76:4eff:fe08:40d8/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether bc:76:4e:08:3d:31 brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.1/24 brd 192.168.3.255 scope global eth2
    inet6 fe80::be76:4eff:fe08:3d31/64 scope link 
       valid_lft forever preferred_lft forever
Enable IP Forwarding

To enable forwarding, you need to enable it in two places. One in /proc/sys/net/ipv4/ip_forward.

echo 1 > /proc/sys/net/ipv4/ip_forward

And the other in /etc/sysctl.conf. The below uses grep check the value of net.ipv4.ip_forward.

grep net.ipv4.ip_forward /etc/sysctl.conf 
net.ipv4.ip_forward = 0

If zero, enable with a one as shown below.

net.ipv4.ip_forward =

1


Configure Static NAT and Forwarding Rules
iptables --table nat --append POSTROUTING --out-interface eth0 -j SNAT --to 95.138.163.75
iptables --append FORWARD --in-interface eth2 -j ACCEPT
service iptables save

We also need to remove the default reject rule on the FORWARD’ing table:

iptables -D FORWARD 1

Here I delete rule number one from the FORWARD table. Make sure you delete the correct line. To see the line numbers, use:

[root@proxy-bast ~]# iptables -vnL --line-number
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1    44444   62M ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           ctstate RELATED,ESTABLISHED 
2        0     0 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0           
3        0     0 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0           
4        1    60 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           ctstate NEW tcp dpt:22 
5        1    40 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited 

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination         
1        0     0 REJECT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-host-prohibited 

Chain OUTPUT (policy ACCEPT 8769 packets, 544K bytes)
num   pkts bytes target     prot opt in     out     source               destination

Make sure you have restarted everything.

service iptables restart
service network restart

Now configure the default gateway on the infrastructure server.

CentOS 7

With the introduction of firewalld, CentOS 7 now does things a little differently.

Method 1

This method uses the predefined zones available to us and is by far the easiest method to apply. The external zone has IP masquerading enabled by default so there should be little to do.

Define Your Zones

To view your zone setup.

[root@proxy-bast ~]# firewall-cmd --get-default-zone
public
[root@proxy-bast ~]# firewall-cmd --get-active-zones
public
  interfaces: eth0 eth1 eth2

To see the supported predefined zones , use the --get-zones</code option.

[root@proxy-bast ~]# firewall-cmd --get-zones
block dmz drop external home internal public trusted work

You can find out more about each zone with the below command.

[root@proxy-bast ~]# firewall-cmd --zone=public --list-all
public (default, active)
  interfaces: eth0 eth1 eth2
  sources: 
  services: dhcpv6-client ssh
  ports: 
  masquerade: no
  forward-ports: 
  icmp-blocks: 
  rich rules:

Or you can list all zones with the --list-all-zones option.

firewall-cmd --list-all-zones

The zones I will be using are external, work and internal.

external
  interfaces: 
  sources: 
  services: ssh
  ports: 
  masquerade:

yes

  forward-ports: 
  icmp-blocks: 
  rich rules:

work
  interfaces: 
  sources: 
  services: dhcpv6-client ipp-client ssh
  ports: 
  masquerade: no
  forward-ports: 
  icmp-blocks: 
  rich rules: 

internal
  interfaces: 
  sources: 
  services: dhcpv6-client ipp-client mdns samba-client ssh
  ports: 
  masquerade: no
  forward-ports: 
  icmp-blocks: 
  rich rules: 

My setup looks like this…

Port	Firewall Zone	Name						IPv4				
------------------------------------------------------------------------
eth0	external		PublicNet (Internet)		162.13.87.197		
eth1	work			ServiceNet (Rackspace)		10.179.198.73		
eth2	internal		Infrastructure				192.168.3.1

…and can be achieved with the below commands. Don’t forget to restart firewalld!

firewall-cmd --permanent --zone=external --change-interface=eth0
firewall-cmd --permanent --zone=work --change-interface=eth1
firewall-cmd --permanent --zone=internal --change-interface=eth2
firewall-cmd --reload
systemctl restart firewalld
Method 2

With this method we use the --direct option so we can include traditional iptable rules.

Enable IP Forwarding

This step is not needed if you are using the predefined “external” zone provided by firewalld, as masquerade is enabled by default already.

echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf

To check its enabled.

[root@proxy-bast ~]# sysctl -p
net.ipv4.conf.eth0.arp_notify = 1
vm.swappiness = 0
net.ipv4.ip_forward = 1
Configure Static NAT and Forwarding Rules
firewall-cmd --permanent --direct --passthrough ipv4 -t nat -I POSTROUTING --out-interface eth0 -j SNAT --to 162.13.87.197
firewall-cmd --permanent --direct --passthrough ipv4 --append FORWARD --in-interface eth2 -j ACCEPT
firewall-cmd --reload

systemctl restart network
systemctl restart firewalld
Method 2

Revert back to the tried and tested iptables.

Revert back to Using IPTables
systemctl stop firewalld
systemctl disable firewalld

iptables-service

touch /etc/sysconfig/iptables
systemctl start iptables
systemctl enable iptables

touch /etc/sysconfig/ip6tables
systemctl start ip6tables
systemctl enable ip6table

Now you can follow the instructions for CentOS 6.6.

Ubuntu 14.04 LTS

In Ubuntu we use the Uncomplicated Firewall (UFW).

Enable IP Forwarding

Use a text editor to open up the below file as root…

nano /etc/default/ufw

…and enable the default forward policy – change to ACCEPT.

DEFAULT_FORWARD_POLICY="

ACCEPT

"

We also need to edit the below…

nano /etc/ufw/sysctl.conf

…and uncomment the following lines.

net/ipv4/ip_forward=1
net/ipv6/conf/default/forwarding=1
Configure Static NAT and Forwarding Rules

As root, open the below file.

nano /etc/ufw/before.rules

From the top, my configuration file looks like the below. I inserted the lines in bold.

#
# rules.before
#
# Rules that should be run before the ufw command line added rules. Custom
# rules should be added to one of these chains:
#   ufw-before-input
#   ufw-before-output
#   ufw-before-forward
#

# nat Table rules *nat :POSTROUTING ACCEPT [0:0] :PREROUTING ACCEPT [0:0] -A POSTROUTING -s 192.168.3.0/24 -o eth0 -j SNAT –to-source 162.13.87.197 -A PREROUTING -i eth2 -j ACCEPT COMMIT

# Don't delete these required lines, otherwise there will be errors
*filter
:ufw-before-input - [0:0]
:ufw-before-output - [0:0]
:ufw-before-forward - [0:0]
:ufw-not-local - [0:0]
# End required lines


# allow all on loopback
-A ufw-before-input -i lo -j ACCEPT

...

You will need to restart ufw for the changes to take effect.

ufw disable && sudo ufw enable

For some reason this wiped my SSH rule:

ufw allow ssh
ufw reload
ufw status verbose

Create Infrastructure Server

Here we spin-up a server connected to our isolated cloud network and no public interface. All communications must go via the proxy-bast server.

supernova uk boot protected --flavor 2 --image 189678ca-fe2c-4b7a-a986-30c3660edfa5 --nic net-id=4d15b8ad-45c5-4169-a4fa-d36f1a776efd --no-service-net --no-public

Configure Internet Gateway

Here we simply need to route the traffic through the proxy bastion. We do this by defining it as our default gateway. We also need to set our DNS servers.

CentOS 6.6

Simplicity!

echo "GATEWAY=192.168.3.1" >> /etc/sysconfig/network
echo "nameserver 83.138.151.80" >> /etc/resolv.conf
echo "nameserver 83.138.151.81" >> /etc/resolv.conf
service network restart

CentOS 7

The default image provided by Rackspace comes with nmcli disabled. As such the process is similar to previous releases.

echo "GATEWAY=192.168.3.1" >> /etc/sysconfig/network
echo "nameserver 83.138.151.80" >> /etc/resolv.conf
echo "nameserver 83.138.151.81" >> /etc/resolv.conf
echo "DNS1=83.138.151.80" >> /etc/sysconfig/network-scripts/ifcfg-eth0
echo "DNS2=83.138.151.81" >> /etc/sysconfig/network-scripts/ifcfg-eth0
systemctl restart network

Ubuntu 14.04 LTS

To define the default gateway, you need to edit the /etc/network/interfaces file.

nano /etc/network/interfaces

Mine looks like this. Make sure to add the gateway.

auto eth0
iface eth0 inet static
    address 192.168.3.4
    netmask 255.255.255.0

gateway 192.168.3.1


You will need to manually add Rackspaces name servers to your resolv.conf. However on Ubuntu this file is automatically generated. Instead we editing /etc/resolvconf/resolv.conf.d/base and regenerate the file using the resolvconf command.

root@protected:~# cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
root@protected:~# echo "nameserver 83.138.151.80" >> /etc/resolvconf/resolv.conf.d/base
root@protected:~# echo "nameserver 83.138.151.81" >> /etc/resolvconf/resolv.conf.d/base
root@protected:~# resolvconf -u
root@protected:~# cat /etc/resolv.conf 
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 83.138.151.80
nameserver 83.138.151.81

I needed to reboot for the changes to take effect.

reboot

Related Documents

Rackspace Developer Blog: Protect your Infrastructure Servers with Bastion Hosts and Isolated Cloud Networks

Rackspace Developer Blog: Supernova: Managing OpenStack Environments Made Easy

Rackspace Knowledge Centre: Using OnMetal Cloud Servers through API

Fedora: Firewalld

Oracle-Base: Linux Firewall (firewalld, firewall-cmd, firewall-config)

Kevin’s Cheat Sheet: Configure iptables to act as a NAT gateway

Rackspace Developer Blog: Getting Started: Using rackspace-novaclient to manage Cloud Servers

James Rossiter: Forward ports in Ubuntu Server 12.04 using ufw

Ubuntu Documentation: Firewall

Github: UFW

Code Ghar: Ubuntu 12.04 IPv4 NAT Gateway and DHCP Server

Linux Gateway: A More Complex Firewall

netfilter.org: Saying How to Mangle the Packets

Ubuntu Documentation: IptablesHowTo

Major.io: Delete single iptables rules

iptables.info: Iptables

snipt.net: Insert an iptables rule on a specific line number with a comment, and restore all rules after reboot

stackexchange.com: How do I set my DNS on Ubuntu 14.04?

thesimplesynthesis.com: How to Set a Static IP and DNS in Ubuntu 14.04

Rackspace Knowledge Centre: Ubuntu – Setup

Rackspace Knowledge Centre: Introduction to iptables

Rackspace Knowledge Centre: Sample iptables ruleset

Ubuntu Geek: Howto add permanent static routes in Ubuntu

NixCraft: Debian / Ubuntu Linux Setting a Default Gateway

Ask Ubuntu: Set up permanent routing (Ubuntu 13.04)

cviorel.com: How to set up a VPN server on Ubuntu

Redhat Support: 10.4. Static Routes and the Default Gateway

Streaming BT Sport on Linux

There are many browser plugins out there which allow us to stream good quality video content, such as Flash and Silverlight. Traditionally we would need to install these manually through Wine – which can get messy!! Unfortunately I need Silverlight in order to watch BT Sport…which is a Microsoft project – so its quite unlikely they’ll be releasing a Linux version in the future 😉

I have recently come across a nifty little program called Pipelight, which takes the pain out of things. It does have a dependency on Wine so, if not already installed, you’ll need to do so. So far I have only tried this on Ubuntu 12.04.

Installing Wine

To install Wine, simply type these commands into you’re Terminal.

 
sudo apt-get update
sudo apt-get install wine

You can configure it with.

winecfg

Installing Pipelight

To be able to stream the BT Sport channels on you’re Ubunutu 12.04 Desktop you will have to install one of the special browser plugins which in our case is the browser plugin Pipelight.
This is key as the Pipelight program allows you to use Windows only plugin inside you’re linux browser.

On Ubuntu 12.04

To install PipeLight on Ubuntu, simply open up you’re terminal and type the following commands.

sudo add-apt-repository ppa:pipelight/stable
sudo apt-get update
sudo apt-get install --install-recommends pipelight-multi
sudo pipelight-plugin --update

After installing piplelight and pipelight-multi they will also install the core fonts for Mircosoft, as Silverlight requires them. If you don’t already have these, during the installion you will be asked to accept a license agreement while pipelight is installing. (just press ‘Enter’ to accept.)

Using and Configuring Plugins

If you just ‘pipelight-plugin’ (or ‘pipelight –help’) at the command line, you will be greeted with a screen like below.

daryl@daryl-Dell-DXP061:~$ pipelight-plugin 

Usage: pipelight-plugin [OPTIONS ...] COMMAND

Environment variables:
  MOZ_PLUGIN_PATH             overwrite plugin path

Options:
  --accept                    accept all licenses

User commands:
  --enable  PLUGIN            enable plugin
  --disable PLUGIN            disable plugin
  --disable-all               disable all plugins
  --list-enabled              list enabled plugins for the current user
  --list-enabled-all          list all enabled plugins
  --system-check              do a system check
  --help                      shows this help
  --version                   shows the current version of Pipelight

Global commands (require root rights):
  --create-mozilla-plugins    create copies of libpipelight.so
  --remove-mozilla-plugins    remove copies of libpipelight.so
  --unlock PLUGIN             unlocks an additional plugin
  --lock   PLUGIN             locks an additional plugin
  --update                    update the dependency-installer script


Supported standard plugins:
  silverlight5.1
  silverlight5.0
  silverlight4
  flash
  unity3d
  widevine

Additional plugins (experimental):
  shockwave
  foxitpdf
  grandstream
  adobereader
  hikvision
  npactivex
  roblox
  vizzedrgr
  viewright-caiway
  triangleplayer
  x64-unity3d
  x64-flash

As you can see, there are a few supported plugins. For now, I am only interested in Silverlight.

You’re plugins by default should all be disable but just incase you can check this by entering the follow command to you’re terminal.

pipelight-plugin --list-enable-all

To install the Silverlight plugin, you simply:

pipelight-plugin --enable silverlight5.0

While i was configuring my plugin’s for the first time, i had enabled silverlight5.1 and was the only plugin i had enabled. I tried then watching BT Sport but failed then had to disable 5.1, as shown below.

pipelight-plugin --disable silverlight5.1

Selection_007

Rescources

Installing Wine

Official Pipelight site:
http://pipelight.net/cms/installation.html