Adventures in VoIP

I haven't posted any telecom stuff in a while, and there are some reasons for that I won't get into. I'm still very much active in the industry.

In the meantime, I've been working on a lot of fun stuff. One of those things was replacing the OS on my older server. I switched from Fedora 22 to Rocky 9 - a massive jump. I made a decision to move most of my applications to Docker, as this was the easiest way to manage the deployment of new applications. My media was a different story, though.

The solution I was previously using implemented a tool called Greyhole to pool drives. Greyhole is a really neat utility, and it has served my purpose well over the years. It works by creating links from a folder called the Landing Zone to files on mounted storage drives. Of course, now I have 60 TB of data on these storage drives, and migrating to a new solution is impractical. I decided to do what any normal person would do and load Greyhole into my new environment. I had no idea what I was in for!

It turns out there are a lot of changes to the underlying Linux operating system between Fedora 22 and Rocky 9. The first major issue for me was that I couldn't use the well documented installation methods to install Greyhole in my environment - that meant building it from scratch.

Creating the Environment

By default Greyhole wants to mount shares in /mnt/samba/ but that wasn't going to work for my setup. I have a /docker/ folder on every server in my network and this is my common point for all services. I created a subfolder in this folder for my Samba shares: /docker/mnt/ with the structure I intend to use for my shares:

/docker└── mnt ├── anime ├── backups ├── books ├── movies ├── music ├── pictures ├── software └── tv

Additionally, Greyhole needs its own folder with a landing zone. The folders in the landing zone will correspond with the Samba shares that will be mounted to /docker/mnt. This is the structure I created:

/greyhole
└── lz
├── anime
├── backups
├── books
├── movies
├── music
├── pictures
├── software
└── tv

Now that I have the correct folder structure, I assigned it to my docker user. I created a docker user and group on each of my servers with the uid and gid of 1500. I have added any users who require access to this folder to the docker group. I ran the chown command and the chmod command to set the ownership and permissions of all the folders:

chown -R /docker/
chmod -R 774 /docker/

The greyhole folder will be owned by root, so we do not need to make any changes to the ownership and permissions of the greyhole folder. Rocky Linux does have other requirements, though - SELinux restricts access

The Back End Database

Greyhole requires a database to work properly. For this implementation I went with MySQL. I installed SQL with these commands:

sudo dnf install -y mysql-server
sudo dnf install -y php-mysqlnd
sudo service mysqld start
sudo systemctl enable mysqld

You can confirm that this has run properly by checking the status of the mysql service:

sudo systemctl status mysql

Next we need to secure mysql. Run this command:

sudo mysql_secure_installation

This will trigger the MySQL Secure Access Wizard. The output below is what I used for my configuration with a dummy root password for public consumption. Typed characters are in BOLD.

MySQL Secure Access
Would you like to setup VALIDATE PASSWORD component? Y
There are three levels of password validation policy:
LOW Length >= 8
MEDIUM Length >= 8, numeric, mixed case, and special characters
STRONG Length >= 8, numeric, mixed case, special characters and dictionary file
Please enter 0 = LOW, 1 = MEDIUM and 2 = STRONG: 0
Please set the password for root here. MySQLr00t
Do you wish to continue with the password provided?(Press y|Y for Yes, any other key for No) : Y
Remove anonymous users? (Press y|Y for Yes, any other key for No) : Y
Disallow root login remotely? (Press y|Y for Yes, any other key for No) : Y
Remove test database and access to it? (Press y|Y for Yes, any other key for No) : Y
Reload privilege tables now? (Press y|Y for Yes, any other key for No) : Y

Now that MySQL is secured, we need to create the user and database that Greyhole will use. First, log in to MySQL. When prompted, enter the root password you set when securing MySQL.

mysql -u root -p

We need to create a greyhole user with a password. The Greyhole documentation suggests using the psasword 89y63jdwe, however this does not meet the modern complexity requirements. For the purpose of this document we will use the password MySql_gr3yhole for the greyhole user. Run these commands to create the greyhole database and user:

CREATE DATABASE greyhole;
CREATE USER 'greyhole_user'@'localhost' IDENTIFIED BY 'MySql_gr3yhole';

Next we need to give the greyhole user permission to access the greyhole database:

USE greyhole;
GRANT ALL PRIVILEGES ON greyhole.* TO 'greyhole_user'@'localhost';

Now that the database is configured, you can exit MySQL by typing exit.

Samba Configuration

Applications on my network will access the files on my server using Samba, and it is necessary for the operation of Greyhole. There are plenty of guides on how to configure Samba in Linux, but I encountered a few things that I didn't expect along the way.

We need to install a Samba client and the CIFS Utilities packages:

sudo dnf install -y cifs-utils samba-client

The smb.conf file will need to be modified to reflect your environment. Here is a snippet of one of the shares I created:

[tv]
path = /greyhole/lz/tv
browseable = yes
writeable = yes
guest ok = no
create mask = 0770
directory mask = 0770
dfree command = /usr/bin/greyhole-dfree
vfs objects = greyhole

Not the last two lines: these are specific to the Greyhole platform. These will need to be added to every share that Greyhole is managing. As we have not yet installed Greyhole, this portion of the Samba configuration file will cause problems. Do not start the Samba service until you have Greyhole installed.

Samba uses its own credentials and user structure for access to resources. In order to create a Samba user you must assign a Samba password to an existing Linux user. In this example, I used the user docker.

sudo smbpasswd -a docker

Enter and confirm the password you want to use for your user. These are the credentials you will use to connect to the Samba shares. Repeat this for any users who you want to give SMB access to your shares.

Samba Configuration: SELinux

SELinux is Linux Kernel module that provides an improved mechanism for securing the file system. I am not going to get into the specific details of SELinux, but I will outline the commands I used and what they mean. At a very high level, there are three portions to an SELinux Security Policy The user, the role, and the type. For this implementation I only used the type.

I need to set the Landing Zone type to be a Samba share. This command will set the context for type to samba_share for all folders under /greyhole/lz/:

sudo semanage fcontext -a -t samba_share_t "/greyhole/lz(/.*)?"

Because the docker folder needs to be accessible to containers we will define the type as a Container File:

sudo semanage fcontext -a -t container_file_t "/docker(/.*)?"

We need to restore the default SELinux permissions now that we have set them. This is done using the restorecon command:

sudo restorecon -Rv /docker/
sudo restorecon -Rv /greyhole/

Finally we need to let SELinux know that we want to allow read/write permissions on all Samba shares. This is done using a boolean option for the entire system - it is possible to restrict this further, but that is outside of the scope of my document.

setsebool -P samba_export_all_rw=1

Building the Storage Pool

I have already touched upon how Greyhole manages storage. We need to mount our storage drives inside the /greyhole/ folder. I have six drives, so I created one folder for each drive. Each physical drive gets mounted to a folder under /greyhole and each drive has a folder called gh in the root. Mount the drives using fstab so that they will automatically mount when the system is booted. The structure of my /greyhole folder now looks like this:

/greyhole/
├── drive1
│ ├── gh
├── drive2
│ ├── gh
├── drive3
│ ├── gh
├── drive4
│ ├── gh
├── drive5
│ ├── gh
├── drive6
│ ├── gh
└── lz
├── anime
├── backups
├── books
├── movies
├── music
├── pictures
├── software
└── tv

If you are deploying a new Greyhole pool, you will need to create the /gh/ folder under each drive manually. If you are migrating an existing drive pool, the folder should already exist.

Manual Installation of Greyhole

I tried to use the Greyhole installation script so many times but it never worked properly for me. Eventually I just gave up and deployed it manually. Note that the file I downloaded was the most current when I wrote this post, but it may have changed - check the releases on Github and grab the latest if it's more current. I've highlighted the relevant lines in green for convenience. It is important to note that the official documentation says to run all these commands as root - so that's exactly what I did. I have also included in this script the command to generate the required Samba modules for SMB 4.20 that are not included as part of the greyhole package.

dnf install -y heimdal-devel
dnf install -y chkconfig
dnf install -y initscripts
dnf install -y lm_sensors-libscd /greyhole
cd /greyhole
wget https://github.com/gboudreau/Greyhole/releases/download/0.15.25/greyhole-0.15.25.tar.gz
tar -xvf greyhole-0.15.25.tar.gz
cd greyhole-0.15.25
GREYHOLE_INSTALL_DIR=`pwd`
mkdir -p /var/spool/greyhole
mkdir -p /var/spool/greyhole
chmod 777 /var/spool/greyhole
mkdir -p /usr/share/greyhole
install -m 0755 -D -p greyhole /usr/bin
install -m 0755 -D -p greyhole-dfree /usr/bin
install -m 0755 -D -p greyhole-php /usr/bin
install -m 0755 -D -p greyhole-dfree.php /usr/share/greyhole
install -m 0644 -D -p schema-mysql.sql /usr/share/greyhole
install -m 0644 -D -p greyhole.example.conf /usr/share/greyhole
install -m 0644 -D -p greyhole.example.conf /etc/greyhole.conf
install -m 0644 -D -p logrotate.greyhole /etc/logrotate.d/greyhole
install -m 0644 -D -p greyhole.cron.d /etc/cron.d/greyhole
install -m 0755 -D -p greyhole.cron.weekly /etc/cron.weekly/greyhole
install -m 0755 -D -p greyhole.cron.daily /etc/cron.daily/greyhole
cp -r web-app /usr/share/greyhole/web-app
install -m 0755 -D -p scripts-examples/greyhole_file_changed.sh /usr/share/greyhole/scripts-examples
install -m 0755 -D -p scripts-examples/greyhole_idle.sh /usr/share/greyhole/scripts-examples
install -m 0755 -D -p scripts-examples/greyhole_notify_error.sh /usr/share/greyhole/scripts-examples
install -m 0755 -D -p scripts-examples/greyhole_send_fsck_report.sh /usr/share/greyhole/scripts-examples
install -m 0755 -D -p scripts-examples/greyhole_sysadmin_notification.sh /usr/share/greyhole/scripts-examples
install -m 0644 -D -p USAGE /usr/share/greyhole
install -m 0755 -D -p build_vfs.sh /usr/share/greyhole
install -m 0644 -D -p docs/greyhole.1.gz /usr/share/man/man1/
install -m 0644 -D -p docs/greyhole-dfree.1.gz /usr/share/man/man1/
install -m 0644 -D -p docs/greyhole.conf.5.gz /usr/share/man/man5/
LIBDIR=/usr/lib
SMB_VERSION="`smbd --version | awk '{print $2}' | awk -F'-' '{print $1}' | awk -F'.' '{print $1,$2}' | tr ' ' .`"
mkdir "$LIBDIR/greyhole"
cp samba-module/bin/$SMB_VERSION/greyhole-$(uname -m).so "$LIBDIR/greyhole/greyhole-samba${SMB_VERSION//.}.so"
install -m 0644 -D -p greyhole.systemd /usr/lib/systemd/system/greyhole.service
cp $GREYHOLE_INSTALL_DIR/samba-module/vfs_greyhole-samba-4.x.c $GREYHOLE_INSTALL_DIR/samba-module/vfs_greyhole-samba-4.20.c
cp $GREYHOLE_INSTALL_DIR/samba-module/wscript-samba-4.x.patch $GREYHOLE_INSTALL_DIR/samba-module/wscript-samba-4.20.patch
$GREYHOLE_INSTALL_DIR/build_vfs.sh current

I know, that's a lot. It can be compiled into a script if you have the knowledge. I only needed to accomplish this once, so I didn't bother scripting it.

This script will copy a sample Greyhole configuration file to /etc/greyhole.conf - you will want to modify this file to reflect your implementation. I have included the relevant entries from my configuration file. Note the db_user and db_pass match what was configured for MySQL. My storage pool drives and Samba shares must all be configured in this section or Greyhole will not work as expected.

db_host = localhost

db_user = greyhole_user

db_pass = MySql_gr3yhole

db_name = greyhole

log_level = DEBUG

log_memory_usage = no

check_for_open_files = yes

num_copies[anime] = 1

num_copies[backups] = 1

num_copies[books] = 1

num_copies[movies] = 1

num_copies[music] = 1

num_copies[pictures] = 1

num_copies[software] = 1

num_copies[tv] = 1

storage_pool_drive = /greyhole/drive1/gh, min_free: 10gb

storage_pool_drive = /greyhole/drive2/gh, min_free: 10gb

storage_pool_drive = /greyhole/drive3/gh, min_free: 10gb

storage_pool_drive = /greyhole/drive4/gh, min_free: 10gb

storage_pool_drive = /greyhole/drive5/gh, min_free: 10gb

storage_pool_drive = /greyhole/drive6/gh, min_free: 10gb

check_storage_pool_schedule = *:00

drive_selection_algorithm = most_available_space

df_cache_time = 15

delete_moves_to_trash = yes

modified_moves_to_trash = yes

ignored_files = \..*\.[0-9a-zA-Z]{6}

ignored_files = [0-9A-F]{8}\.tmp

ignored_files = \.cprestoretmp.*

ignored_folders = .*/_UNPACK_.*

ignored_folders = .*/inProgress/.*

ignored_folders = .*/_unpack.*

ignored_files = \.DS_Store

Fixing the Greyhole Implementation

After we have completed the Greyhole installation, we need to fix it or it will never start! Greyhole requires MySQL and Samba to be running. The check for MySQL is great, but the check for Samba refers to the requirement as smbd - Rocky only uses smb. Edit your /usr/lib/systemd/system/greyhole.service file using your favourite editor and change the requirement from smbd to smb:

Requires=mysqld.service smb.service

Mounting Shares Locally

There is a documented method to create a service that will mount shares locally on the server so that applications can write files that Greyhole will be able to see. The exact script didn't fit my requirements, so I made a few changes. The first thing we need to do is create a credentials file for our Samba user. This is a plain text file that contains the Samba username and password that Greyhole will use to mount the shares. I put my in my /greyhole/ folder and called it .smb_credentials. Using your favourite text editor, populate the file with a username and password. My sample file is below:

username=docker
password=**********

Now we need to create the mount_shares_locally service. Using your favourite text editor, create the file /etc/init.d/mount_shares_locally and modify the contents of the file to reflect the Greyhole configuration you are implementing. Mine is included below. For convenience, I have identified the specific sections I changed.

#!/bin/bash
#
### BEGIN INIT INFO
# Provides: mount_shares_locally
# Required-Start: $network $local_fs $remote_fs smb mysqld
# Required-Stop: $network $local_fs $remote_fs smb
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: mount Samba shares locally
### END INIT INFO

username="docker"

if [ -f /etc/rc.d/init.d/functions ]; then
. /etc/rc.d/init.d/functions
fi
start () {
uid=`id -u $username`
gid=`id -g $username`

echo -n $"Mounting Samba shares locally: "
mkdir -p /docker/mnt/
cd /docker/mnt
testparm -s /etc/samba/smb.conf 2>/dev/null | grep "^\[" | grep -v "\[global\]\|\[homes\]\|\[netlogon\]\|\[sysvol\]\|\[printers\]" | awk -F'[' '{print $2}' | awk -F']' '{print $1}' | xargs -d "\n" mkdir -p
sleep 5 opt="credentials=/greyhole/.smb_credentials,uid=${uid},gid=${gid},file_mode=0660,dir_mode=0770,nobrl,hard,_netdev,iocharset=utf8,noserverino,mfsymlinks"
if mount.cifs 2>&1 | grep vers= >/dev/null; then
opt="$opt,vers=3.0"
elif man mount.cifs 2>&1 | grep vers= >/dev/null; then
opt="$opt,vers=3.0"
fi
while IFS='' read -r d; do
if [ "`mount | grep "//127.0.0.1/$d/* on " | wc -l`" = "0" ]; then
/sbin/mount.cifs "//127.0.0.1/$d" "$d" -o $opt
else
echo " Share [$d] is already mounted."
fi
done < <(testparm -s /etc/samba/smb.conf 2>/dev/null | grep "^\[" | grep -v "\[global\]\|\[homes\]\|\[netlogon\]\|\[sysvol\]\|\[printers\]" | awk -F'[' '{print $2}' | awk -F']' '{print $1}')
touch /var/lock/subsys/mount_shares_locally
success $"$base startup"
echo
return 0
}

stop () {
echo -n $"Unmounting locally mounted Samba shares: "
/bin/umount -l /docker/mnt/*
rm -f /var/lock/subsys/mount_shares_locally
success $"$base shutdown"
echo
return 0
}

restart () {
stop
start
}

case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)
echo $"Usage: $0 {start|stop|restart}"
exit 1
;;
esac
exit $?

Now that the file has been created, we need to make it a functional script. First we change the permissions to allow it to be executed, then we load it into the Linux operating system.

sudo chmod +x /etc/init.d/mount_shares_locally
sudo systemctl enable mount_shares_locally

Bringing it all together

Let's recap. Here is what we have accomplished so far:

Created our folder structure
Installed MySQL
Configured Samba
Configured SELinux to allow our shares to work
Built a storage pool
Manually installed Greyhole
Modified the Greyhole configuration to suit our environment

There isn't much left to do at this point. We have completed all of the configuration that was required, and now we just have to start everything. We will need to reload the System Daemon to recognise the changes we made to the greyhole service and the mount_shares_locally service.

sudo systemctl daemon-reload

Now we can load Samba and Greyhole.

sudo service smb start
sudo service greyhole start

If you have done this properly, Greyhole should copy anything that hits your Samba Shares (locally at /docker/mnt or remotely at //<<server-ip-or-fqdn>>/<<share-name>>) to the drive pool. You can check the status of your drive pool by running greyhole -s. Here is my output:

#greyhole -s

Greyhole Statistics
===================

Storage Pool
Total - Used = Free + Trash = Possible
/greyhole/drive1/gh: 5544G - 2479G = 2786G + 67G = 2853G
/greyhole/drive2/gh: 5081G - 2039G = 2785G + 12G = 2798G
/greyhole/drive3/gh: 7392G - 4242G = 2777G + 64G = 2841G
/greyhole/drive4/gh: 14783G - 11253G = 2786G + 66G = 2852G
/greyhole/drive5/gh: 16696G - 13075G = 2783G + 82G = 2865G
/greyhole/drive6/gh: 18551G - 14835G = 2785G + 36G = 2822G
==========================================
Total: 68048G - 47922G = 16703G + 327G = 17030G

If you are migrating an existing pool, you will want to run a full fsck with the options to delete orphaned metadata, to find orphaned files, and to fix any broken symlinks.

sudo greyhole --fsck --delete-orphaned-metadata --find-orphaned-files --fix-symlinks

Let me know if you are able to replicate my results. I did A LOT of stuff that isn't included in this guide, but I've pared it down to just what I think I need to include. If you are having difficulty, let me know. We may be able to set up a working session to figure it out together, then I can update my docs.

Adventures in VoIP

Installing and configuring Greyhole in Rocky Linux 9

Creating the Environment

The Back End Database

Samba Configuration

Samba Configuration: SELinux

Building the Storage Pool

Manual Installation of Greyhole

Fixing the Greyhole Implementation

Mounting Shares Locally

Bringing it all together

NRS Backup location

Voicemail Pro Automatic Service Restart

Automatic Daylight Savings Time Update

Total Pageviews

Translate