Clam Anti Virus on Arch Linux

Install Clam with:

sudo pacman -Sy clamav

To update the virus definitions, run:

sudo freshclam

You will also need to start and enable the clamam-freshclam.service to get the latest definitions at boot.

sudo systemctl enable --now clamav-freshclam.service

Make sure you have already run sudo freshclam before starting and enabling the clamav-daemon. If not, stop it and run.

sudo systemctl enable --now clamav-daemon.service

It is recommended to add the additional signatures from the AUR package, clamav-unofficial-sigs:

yaourt -Sy clamav-unofficial-sigs

Make sure the below is uncommented:

[andy@work-pc ~]$ grep user_configuration_complete /etc/clamav-unofficial-sigs/user.conf
user_configuration_complete="yes"

Then to install and enable these unofficial signatures:

sudo clamav-unofficial-sigs.sh --install-all

To update the unofficial signatures, run:

sudo clamav-unofficial-sigs.sh

Lastly, scan for viruses:

mkdir ~/clamav-scan-results/
clamscan --recursive --infected --exclude-dir='^/sys|^/dev|^/proc|^/var/lib/clamav' --max-filesize=4000M --max-scansize=4000M / -l ~/clamav-scan-results/201803261436

Apache not Logging Correct IP when behind Incapsula WAF

Incapsula is a great resource to help protect your web site from unwanted traffic and attacks. It is a cloud-based application delivery platform, providing among other things:

  • Content Delivery Network (CDN)
  • Distributed Denial of Service (DDoS) Mitigation
  • Web Application Firewall (WAF)

Incapsula acts as a proxy, sitting in front of the nodes its protecting. The DNS points to Incapsula which hides the IP address to your site.  Incapsula analyses the traffic and removes any unwanted requests before passing it on to the web node.

As with any proxy-based system, the proxy rewrites the the X-Forwarded-For header information with the originating IP address.  However, Apache needs to be configured to use the header information.

Enable X-Forwarded-For

To enable X-Forwarded-For, open the main Apache configuration file and find the section that defines the LogFormat:

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

Then add the following additional line:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy

Lastly edit the configuration file for your virtual host:

# vim /etc/apache2/sites-enabled/pikedom.com.conf

Then comment out the existing CustomLog, combined in my example:

#CustomLog /var/www/pikedom.com/pikedom-access.log combined

And add a new entry for the CustomLog we created, proxy:

CustomLog /var/www/pikedom.com/pikedom-access.log proxy

Check Apache configuration for errors:

# apachectl -t

If none, restart Apache:

# service apache2 restart

To confirm X-Forward-For is working, first confirm what your public IP address is:

[andy@home-pc ~]$ curl -4 icanhazip.com
180.112.113.2

Then tail the access log and grep for your IP while visiting the site:

root@webhost1:~# tailf /var/www/pikedom.com/pikedom.com-access.log | grep 180.112.113.2
180.112.113.2 - - [26/Mar/2018:10:39:02 +0100] "GET / HTTP/1.1" 301 325 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"
180.112.113.2 - - [26/Mar/2018:10:39:02 +0100] "GET / HTTP/1.1" 200 17576 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"
180.112.113.2 - - [26/Mar/2018:10:39:03 +0100] "GET /skin/frontend/pikedom/default/favicon.ico HTTP/1.1" 200 1243 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0"
^C
root@webhost1:~#

Job done!

MS SQL Maintenance Plan not Deleting Previous Backups

I don’t normally include Windows related stuff here because…….well…..its Windows! But this one irritated me so much, I thought I’d share in case anyone else was experiencing this problem.

So I’m using SQL server 2012 to backup all my databases. I have a daily backup running every night at midnight. I also have a maintenance plan in place to delete backups older than 3 days. The backup has been working fine but the problem that the maintenance plan was not deleting any old backups. Frustratingly there was not much to go on in the logs.

Below are my successful settings.

Backup Plan
Backup Plan
Maintenance Plan
Maintenance Plan

The important part here was to make sure the verify backup integrity was ticked!

Install Ansible on Arch Linux

Installation is pretty simple….

$ sudo pacman -Sy ansible

Create an inventory hosts file:

$ sudo vim /etc/ansible/hosts

My one currently has just localhost as the control machine:

[control]
127.0.0.1

Test it works by using the ansible ping module:

[andy@home-pc ~]$ ansible control -m ping -u andy --ask-pass
SSH password: 
127.0.0.1 | SUCCESS => {
    "changed": false, 
    "ping": "pong"
}

To use the –ask-pass switch, you need to first install sshpass:

$ sudo pacman -Sy sshpass

Other useful dependencies are:

Optional dependencies for ansible
    sshpass: for ssh connections with password
    python2-passlib: crypt values for vars_prompt
    python2-netaddr: for the ipaddr filter
    python2-systemd: log to journal
    python2-pywinrm: connect to Windows machines
    python2-dnspython: for dig lookup
    python2-ovirt-engine-sdk: ovirt support
    python2-boto: aws_s3 module
    python2-jmespath: json_query support

Its probably best to enable these as and when they’re needed. One of my next aims to is connect to a Windows device to run a few tasks so I know I’m going to need the winrm module. The below playbook refreshes the pacman cache and then updates the system. It then installs the python2-pywinrm package.

$ vim ~/ansible
$ cd ~/ansible
$ vim winrm.yml

winrm.yml:

---
- name: All hosts up-to-date
  hosts: control
  remote_user: root
  become: yes
  
  tasks:
    - name: full system upgrade
      pacman:
        update_cache: yes
        upgrade: yes
    - name: ansible winrm module
      pacman:
        name: python2-pywinrm
        state: latest

Then run it with:

$ ansible-playbook --ask-pass winrm.yml

Enter the root password.

XWiki on Ubuntu 16.04 LTS with Nginx Reverse Proxy

Install XWiki and all dependant programs on a 4 GB cloud server. This means:

1) Java
2) Tomcat
3) MySQL/MariaDB
4) XWiki
5) Nginx

Here we use Nginx as a reverse proxy to firstly redirect all HTTP to HTTPS and then forward all requests on port 80/443 to port 8080 (tomcat) on the localhost. Here I use LetsEncrytp for my SSL certificates.

Before we begin…

Setup DNS

wiki.dummydomains.org.uk ——> 134.213.27.60

Prepare the Server

Update

Update and reboot the server.

apt-get update
apt-get dist-upgrade
reboot

Enable the firewall

ufw status
ufw enable
ufw allow ssh
ufw reload
ufw status

Install Oracle Java

This is a requirement before installing Tomcat or XWiki. At the time of writing, Java 8.x is recommended as 9.x is too new and has a number of known bugs still.

apt-get install software-properties-common
add-apt-repository ppa:webupd8team/java
apt-get update
apt-get install oracle-java8-installer

You will need to accept the license agreement:

Accept license agreement

Binary code license terms

Because many programs check for $JAVA_HOME, it is a good idea to set it now. If you don’t know the path, check with:

root@wiki:~# update-alternatives --config java
There is 1 choice for the alternative java (providing /usr/bin/java).

  Selection    Path                                     Priority   Status
------------------------------------------------------------
  0            /usr/lib/jvm/java-8-oracle/jre/bin/java   1081      auto mode
* 1            /usr/lib/jvm/java-8-oracle/jre/bin/java   1081      manual mode

Press <enter> to keep the current choice[*], or type selection number:

Then edit your system $PATH variable so that the /usr/lib/jvm/java-8-oracle is the first path.

nano /etc/environment

Mine looks like this:

root@wiki:~# cat /etc/environment
PATH="/usr/lib/jvm/java-8-oracle:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"

You will need to log out and back in first but you can test with the below command.

root@wiki:~# echo $JAVA_HOME
/usr/lib/jvm/java-8-oracle

Create Virtual Host and Generate SSL

Install Nginx and LetsEncrypt.

Nginx

Install and configure Nginx.

apt-get install apache2-utils nginx
systemctl enable nginx

Create a very basic virtual host by editing the nginx configuration file and inserting your server name in the server_name variable.

vim /etc/nginx/sites-enabled/default

Mine looks like this.

root@wiki:~# egrep -v "^$|^[[:space:]]*#" /etc/nginx/sites-available/default 
server {
	listen 80 default_server;
	listen [::]:80 default_server;
	root /var/www/html;
	index index.html index.htm index.nginx-debian.html;
	server_name wiki.dummydomains.org.uk;
	location / {
		try_files $uri $uri/ =404;
	}
}

Restart:

systemctl restart nginx

Check it works!

Nginx test page
If it doesn’t, check the firewall…

Allow HTTP and HTTPS Traffic

If you use a local firewall like UFW or iptables, you will need to allow port 80 and 443.

ufw status
ufw allow http
ufw allow https
ufw status
ufw reload

LetsEncrypt

add-apt-repository ppa:certbot/certbot
apt-get update
apt-get install python-certbot-nginx
certbot --nginx -d wiki.dummydomains.org.uk -d dummydomains.org.uk

Your certificate will get saved to /etc/letsencrypt/live/wiki.dummydomains.org.uk.

LetsEncrypt will edit your virtual hosts file.  The parts we are interested in are:

  listen 443 ssl; # managed by Certbot
  ssl_certificate /etc/letsencrypt/live/wiki.dummydomains.org.uk/fullchain.pem; # managed by Certbot
  ssl_certificate_key /etc/letsencrypt/live/wiki.dummydomains.org.uk/privkey.pem; # managed by Certbot
  include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

  if ($scheme != "https") {
    return 301 https://$host$request_uri;
  } # managed by Certbot

 

Install XWiki

Install XWiki.

wget -q "https://maven.xwiki.org/public.gpg" -O- | apt-key add -
wget "https://maven.xwiki.org/stable/xwiki-stable.list" -P /etc/apt/sources.list.d/
apt-get update

Search for XWiki packages to install.

apt-cache search xwiki

According to the official documentation, the enterprise version is out-of-date and the non-enterprise version should be used.

apt-get install xwiki-tomcat8-mysql

Set the root MySQL password:

2@&EG7dMhPF^44ed

Set root MySQL password
Set root MySQL password

When asked if you should configure the database with dbconfig-common, say yes.

Configure with dbconfig-common
Configure with dbconfig-common

MySQL application password:

zB9j@xht4@

Select application password
Select application password

Check tomcat8 is listening on port 8080:

root@wiki:~# netstat -plnt | grep :8080
tcp6       0      0 :::8080                 :::*                    LISTEN      15840/java

Check your memory usage:

root@wiki:~# free -h
              total        used        free      shared  buff/cache   available
Mem:           3.9G        487M        1.9G         10M        1.5G        3.3G
Swap:            0B          0B          0B

You will need to increase the default about of memory allocated to Java. Here’s how:

vim /etc/default/tomcat8

Before:

root@wiki:~# grep ^JAVA_OPTS /etc/default/tomcat8
JAVA_OPTS="-Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC"

After:

root@wiki:~# grep ^JAVA_OPTS /etc/default/tomcat8
JAVA_OPTS="-Djava.awt.headless=true -Xmx1024m -XX:+UseConcMarkSweepGC"

Restart Tomcat

systemctl restart tomcat8

Test using using a browser:

http://dummydomains.org.uk:8080/xwiki

If you’ve enabled a firewall and you want to test:

ufw allow 8080/tcp
ufw reload

However I’m not going to do this – I’m going to setup Nginx as a proxy first.

Configure Nginx

Remove the default virtual host configuration.

rm -v /etc/nginx/sites-enabled/default.conf
vim /etc/nginx/sites-available/wiki.dummydomains.org.uk.conf

My site configuration look as follows:

upstream tomcat {
  server 127.0.0.1:8080 fail_timeout=0;
  keepalive 64;
}

server {
  
  listen 134.213.27.60:80;
  listen [::]:80;
  
  listen 134.213.27.60:443 ssl;
  listen [::]:443 ssl;
  
  server_name wiki.dummydomains.org.uk dummydomains.org.uk;
  
  ssl_certificate /etc/letsencrypt/live/wiki.dummydomains.org.uk/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/wiki.dummydomains.org.uk/privkey.pem;
  include /etc/letsencrypt/options-ssl-nginx.conf;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
  
  # redirect http to https
  if ($scheme != "https") {
    return 301 https://$host$request_uri;
  }

  auth_basic "Authentication Required";
  auth_basic_user_file xwiki-access;

  location / {
    client_max_body_size 20M;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass_request_headers on;
    proxy_set_header Connection "keep-alive";
    proxy_store off;
    proxy_headers_hash_max_size 512;
    
    allow 81.143.223.118;
    allow 188.114.113.2;
    deny all;

    proxy_pass http://tomcat/;
  }
}

I also want to password protect my wiki:

htpasswd -c /etc/nginx/xwiki-access andy

Enable the site:

cd /etc/nginx/sites-enabled/
ln -s ../sites-available/xwiki.dummydomains.org.uk.conf .

….and check configuration file for errors.

root@wiki:~# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Check config and restart Nginx:

systemctl restart nginx

Now try entering the following into your browser and complete the on-screen installation instructions:

https://wiki.dummydomains.org.uk/xwiki

Complete Installation

Here are few things I normally do after an installation.

Make Root Application

I want to make this Wiki instance the root web application and remove the trailing /xwiki from the URL.

systemctl stop tomcat8.service
mv -v /etc/tomcat8/Catalina/localhost/xwiki.xml /etc/tomcat8/Catalina/localhost/ROOT.xml
vim /etc/xwiki/xwiki-tomcat8.xml

Change:

<Context path="/xwiki" docBase="/usr/lib/xwiki" privileged="true" crossContext="true">
  <!-- make symlinks work in Tomcat -->
  <Resources allowLinking="true" />
</Context>

To:

<Context path="/" docBase="/usr/lib/xwiki" privileged="true" crossContext="true">
  <!-- make symlinks work in Tomcat -->
  <Resources allowLinking="true" />
</Context>

Don’t forget to start Tomcat again:

systemctl start tomcat8.service

Now the URL is simply:

https://wiki.dummydomains.org.uk

Enable superadmin

This is needed if you plan to import XWiki pages from a previous installation.

vim /etc/xwiki/xwiki.cfg

Find the following section.

#-# Enable to allow superadmin. It is disabled by default as this could be a
#-# security breach if it were set and you forgot about it. Should only be enabled
#-# for recovering the Wiki when the rights are completely messed.
# xwiki.superadminpassword=system

….and change to:

#-# Enable to allow superadmin. It is disabled by default as this could be a
#-# security breach if it were set and you forgot about it. Should only be enabled
#-# for recovering the Wiki when the rights are completely messed.
xwiki.superadminpassword=siFwMXlUzKQ6

Don’t forget to restart Tomcat if necessary.

Update Cookie Encryption Keys

When a user logs in, three cookies are saved to their machine. These cookies are encrypted with the below details. First we need to get the two random strings of equal length.

root@wiki:~# date +%s | sha256sum | base64 | head -c 32 ; echo
MWJjNzE4ZTE2ODM0MTVlZDNjODVmNjJl
root@wiki:~# date +%s | sha256sum | base64 | head -c 32 ; echo
Y2M5M2M2ZGEyMGRkYzM3ZmJjZTYyNjYy

Then edit the xwiki.cfg file.

vim /etc/xwiki/xwiki.cfg

Find the relevant section and edit to look like the below.

xwiki.authentication.validationKey=MWJjNzE4ZTE2ODM0MTVlZDNjODVmNjJl
xwiki.authentication.encryptionKey=Y2M5M2M2ZGEyMGRkYzM3ZmJjZTYyNjYy

Don’t forget to restart Tomcat if necessary.

Complete the Installation

Login to complete the installation.

Log in
Log in

Click continue.

Installation wizard
Installation wizard

Register and log in.

Register and login
Register and login

Continue.

Install xwiki
Install xwiki

Select 9.9 and continue.

Install 9.9
Install 9.9

Confirm installation again.

Confirm installation
Confirm installation

Continue.

Continue
Continue

Continue again.

Continue again
Continue again

Confirm the report by clicking continue.

Confirm installation report
Confirm installation report

Installation complete!

Installation Complete
Installation Complete

Import old XWiki Content

Lets see if the import feature works!  Log in as the superadmin user and then navigate to the Administration section:

https://wiki.dummydomains.org.uk/bin/admin/XWiki/XWikiPreferences

Then select Content, followed by Import:

Import
Import

Select the backup.xar that you (hopefully) took earlier and import all the content.

Package content
Package content

Select the following options.

Import options
Import options

 

Installing tox on Manjaro i3

Tox is an open source secure alternative to the likes of Skype. To install it, you need to install the core package and one of the available GUIs.  You can compare some of the available clients here but personally I like qtox. The below will pull in the required dependencies.

sudo pacman -Sy qtox

If you want the latest Git version, you can install qtox-git from the AUR.

yaourt -S qtox-git

Make sure you run that as a regular user – not root.  This pulls in the core package from the ABS repository also.

To add a contact, you will need their Tox ID – which looks like this:

56A1ADE4B65B86BCD51CC73E2CD4E542179F47959FE3E0E21B4B0ACDADE51855D34D34D37CB5

If that is a bit too annoying for you, you can use a free ToxDNS provider, such as utox.org. This will give you a “username@utox.org” address to give your people.

How to Safely Reduce the Size of a Logical Volume

Do not attempt to shrink a volume if the partition is mounted!  Always unmount first! If it is the root volume you need to reduce in size, use a Live DVD/USB instead. You should also take a backup first too 😉

So here we can see I have one Physical Volume, /dev/md0, attached to my one volume group, RAIDVG.

[andy@home-pc ~]$ sudo pvs
  PV         VG     Fmt  Attr PSize PFree
  /dev/md0   RAIDVG lvm2 a--  1.91t    0

We can also see there is zero space left on the volume group.  You can use sudo pvdisplay to see a more detailed output.

As we can see, all this space is being used up by two Logical Volumes.

[andy@home-pc ~]$ sudo lvs
  LV        VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  MediaLV   RAIDVG -wi-ao----   1.32t                                                    
  StorageLV RAIDVG -wi-ao---- 600.00g

I can see the filesystem reports I have 664G available.  I am going to play it nice and safe and only attempt to reduce the size of my LV by 400G.

[andy@home-pc ~]$ sudo df -h /dev/RAIDVG/MediaLV 
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/RAIDVG-MediaLV  1.4T  603G  664G  48% /plex

Don’t forget to unmount!

[andy@home-pc ~]$ sudo umount -v /dev/RAIDVG/MediaLV
umount: /plex (/dev/mapper/RAIDVG-MediaLV) unmounted

Check for potential issues.

[andy@home-pc ~]$ sudo e2fsck -f /dev/RAIDVG/MediaLV
e2fsck 1.43.4 (31-Jan-2017)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/RAIDVG/MediaLV: 35668/88866816 files (11.1% non-contiguous), 163854225/355446784 blocks

It is very important that you run the --reseizefs operand at the same time.

[andy@home-pc ~]$ sudo lvreduce -L -400G /dev/RAIDVG/MediaLV --resizefs
fsck from util-linux 2.29.2
/dev/mapper/RAIDVG-MediaLV: clean, 35668/88866816 files, 163854225/355446784 blocks
resize2fs 1.43.4 (31-Jan-2017)
Resizing the filesystem on /dev/mapper/RAIDVG-MediaLV to 250589184 (4k) blocks.
The filesystem on /dev/mapper/RAIDVG-MediaLV is now 250589184 (4k) blocks long.

  Size of logical volume RAIDVG/MediaLV changed from 1.32 TiB (347116 extents) to 955.92 GiB (244716 extents).
  Logical volume RAIDVG/MediaLV successfully resized.

Note the (minus) -400G. This means reduce by 400G. If I had used 400G instead, LVM would have made the LV 400G, reducing it from 1.3T.  I would have lost over 200G of actual data. Ouch!

And finally I now have 400G available in my VG.

[andy@home-pc ~]$ sudo pvs
  PV         VG     Fmt  Attr PSize PFree  
  /dev/md0   RAIDVG lvm2 a--  1.91t 400.00g

Be careful and remember to take a backup!

Windows 7 QEMU Guest

Check your kernel was compiled with support.

zgrep CONFIG_KVM /proc/config.gz
zgrep VIRTIO /proc/config.gz

Check the kvm and virtio kernel modules are loaded.

lsmod | grep kvm
lsmod | grep virtio

I needed to manually load the virtio module:

sudo modprobe virtio

To automatically load the virtio module at boot:

echo "virtio" >> /etc/modules-load.d/virtio.conf

Reboot and check again:

lsmod | egrep 'virtio|kvm'

Install qemu. I also installed qemu-launcher – a GUI front-end.

sudo pacman -S qemu qemu-launcher

Create a working directory:

mkdir vms
cd vms

Create a virtual hard drive the the Windows installation.

qemu-img create -f qcow2 windows.qcow2 40G

Prepare the installation medium. I inserted a Windows 7 installation DVD and created an ISO locally.

sudo dd if=/dev/sr0 of=en-windows-7-professional-x64-dvd.iso

To launch into the Windows installation:

qemu-system-x86_64 -enable-kvm -m 4096 -cdrom en-windows-7-professional-x64-dvd.iso -boot d windows.qcow2

Once installed you don’t need to attach the ISO:

qemu-system-x86_64 -enable-kvm -m 4096 -boot d windows.qcow2 

Update and upgrade to Windows 10 😉

Resources

https://wiki.archlinux.org/index.php/QEMU

https://www.reddit.com/r/archlinux/comments/1fg3y9/guide_to_running_windows_7_in_qemu/

Spotify on Manjaro i3

Spotify is available from the AUR.

Install

yaourt -S spotify

To play local file you will need to also install ffmpeg.

yaourt -S ffmpeg0.10

Issues

For some (currently) unknown reason, launching Spotify crashes when I login. The only fix I have found so far is to change the scale factor from the default 1 to something else. You can do that by launching it at the command line:

spotify --force-device-scale-factor=2

To make this permanent, edit /usr/share/applications/spotify.desktop.

[Desktop Entry]
Name=Spotify
GenericName=Music Player
Comment=Spotify streaming music client
Icon=spotify-client
Exec=spotify --force-device-scale-factor=2
TryExec=spotify
Terminal=false
Type=Application
Categories=Audio;Music;Player;AudioVideo;
MimeType=x-scheme-handler/spotify;

Resources

https://wiki.archlinux.org/index.php/spotify