NRS Backup location

 Any time I've had to recover a signaling server the biggest concern is extracting the NRS database and recovering it. The NRS backup file can be pulled as long as there is access to the file system.

The file is located at /var/opt/nortel/sps/backup/nrsback.tar

Once the NRS file is in hand the rest is easy.

Voicemail Pro Automatic Service Restart

Today someone asked me if Voicemail Pro on Linux would attempt an automatic restart if the service failed, as it would in Windows. Unfortunately the answer is a resounding NO!

I've been using Linux for years and I didn't see any technical reason why this couldn't be done. I decided to do some digging, and I believe that by making the changes I outline below to the Voicemail Pro service you should be able to have it automatically restart in the event of an abnormal termination. 

DISLCLAIMER: I have not tested this in a production environment. This is not supported by Avaya. If you need this functionality you can implement this fix at your own risk. I suggest opening a GRIP Request with Avaya to have them add this functionality to a new release! 

First, you will need to log in to the Application Server as the root user. Once logged in the next step is to edit the Voicemail Pro service. Enter the following command:

nano /etc/systemd/system/vmpro.service

This will launch the GNU nano editor, allowing you to modify the service directly. The image shown here is the configuration before any changes are made.

Use the arrow keys to navigate to the Service section. In order to tell the service to automatically restart we need to add the following lines:

Restart=on-failure

RestartSec=30s

StartLimitIntervalSec=200

StartLimitBurst=5

This tells the service to restart on failure after waiting 30 seconds, to a maximum of five times. 

Once the changes have been made, use <CTRL>+O to write the file, then <CTRL>+X to exit nano.Once back at the command prompt the system needs to know that you've made changes. Enter the following command:

systemctl daemon-reload

Once this is done the Voicemail Pro service may need to be restarted (again, this is untested), so go ahead and restart the service. 

These instructions should work on any Linux-based IP Office release to date, however future releases may include an updated version of systemd, the process that controls Linux services. If Avaya includes a newer version of systemd these instructions will need to be modified. 

If you do try to implement this, please let me know how well it worked in the comments.

Automatic Daylight Savings Time Update

 The IP Office has the capacity to hold automatic daylight savings time entries for up to ten years. This may sound like a lot, but I'm starting to run into systems that have not had their automatic daylight savings times updated, and are failing to present the correct time as a result. It's easy enough to edit the entries in the list, however this can be tedious - especially if you are responsible for managing a bunch of sites. To address this, I have created a CSV file that contains the next decade of clock changes. I hope that the time change is abolished before I need to make a new one, though!

You can download the file here. Once downloaded, you need to remove the existing entries from your automatic DST settings and then import the file.

Step 1: Removing the existing entries

Log in to the IP Office using IP Office Manager. The Time Settings can be found under System --> System. Locate the drop-down box and select Delete for each entry. This may include some entries from the future, but don't worry. We will import the full decade's worth of entries. 

Once all the entries have been deleted merge the changes back to the system. From there you will need to import the CSV file you downloaded earlier. Select File --> Import/Export --> Import. This will open the Import Configuration window. The default folder will be a subfolder of your Manager folder, but I like to use a folder specific to the restoration process. In this example I will put the csv file in C:\temp and use that as my folder. Put the CSV file in place first, then select the folder.

Select the check box for Configuration. If you have followed all the steps correctly there should be no other options. Click OK to import the new time change profiles. Once imported, check in Manager to see if the profiles are there:

If everything looks right you can click OK on the System record and merge the changes. If you still need help getting the correct time set, please see my post about changing the time and date on an IP Office.

Adding a CertBot Certificate to IP Office

The IP Office generates a self-signed certificate for the purpose of presenting its identity. Any Avaya elements will trust this certificate, but modern web browsers will not by default. One option we have is to add the Avaya Certificate Authority to our list of trusted issuers, but a better way is to ensure that the IP Office presents a certificate that is universally trusted. The Electronic Frontier Foundation (EFF) is providing free publicly trusted certificates with a three month expiration. The short duration is intentional - any malicious use of the certificates wouldn't last longer than three months, making the Internet a safer place. 

The EFF has a utility called CertBot that can be used to generate these certificates on your local machine. They use Public DNS to validate that the correct machine is requesting the certificate. The whole process takes only a few minutes, and when run on a web server can automatically apply the certificates to the hosted pages.Unfortunately Avaya has not yet integrated CertBot functionality into their finished product, so we will need to manually create a certificate, extract the portion that the IP Office needs, and apply the certificate. This will need to be done on a regular basis because of the short certificate duration. Please reach out to your Avaya partner or account manager and ask them to add CertBot integration to the IP Office in future releases! As more people request this functionality Avaya becomes more likely to include it in a future software release. You can download CertBot here. We can't install CertBot directly on the IP Office, so a Linux PC or server will be required. Once CertBot is installed on your Linux PC or Server we can proceed with generating a new certificate. 

The next step is to use CertBot to create a certificate. For the purpose of this example I will redact all my screenshots, hiding the actual FQDN I used. For reference I will use the domain ipo.adventuresinvoip.ca in all of the commands. Simply change out the FQDN for the FQDN you want to assign to your IP Office. Since the CertBot utility uses DNS records you need to be sure to have a DNS A Record for the FQDN you are using. This should point to the public IP Address of the router that the Linux machine is connected to. You will need to enable port forwarding for Port 80 to the Linux machine to get CertBot to work properly since it checks for an HTTP request/response. 

My Linux machine already has a web server running, so I chose to leverage that for the generation of the certificate. To generate the certificate I ran the following command:

certbot run --domains ipo.example.ca

If you don't have a web server running on your Linux machine you can run the commmand in Certificate Only mode:

certbot certonly --domains ipo.example.ca

Once the certificate chain has been created you may be prompted to install it on your web server - since we aren't generating the certificate for this machine we should not be applying it to this server. 

Now we have a new certificate. It will be located in the CertBot folder under a subfolder for that FQDN. In our example this full path would be:

/etc/letsencrypt/live/ipo.example.ca/

The IP Office is unable to use the certificate in this format so we will need to convert it. Change to the folder containing the certificate and extract the PKCS#12 Certificate from the full chain with these commands:

cd /etc/letsencrypt/live/ipo.example.ca/
openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out ipo.p12 -CAfile chain.pem

The second line will prompt for an export password. This password will be used to add the certificate back to the IP Office. Once complete the file ipo.p12 will be created. We can apply this certificate to the IP Office. Copy the ipo.p12 file back to your local PC and launch a web browser. The certificate is easiest to apply using Web Manager, so that's the method I will cover here. 

Log in to Web Manger using the Administrator account (other accounts can be used as long as the permissions allow the manipulation of certificates). Select Security from the top menu, then Certificates. 

This will take you to the Certificates portion of IP Office Web Manager. Here you will see a list of all the trusted certificate authorities, as well as some options to manipulate the IP Office certificate. 

The button for Set will allow us to upload our new certificate. Click Set then locate the ipo.p12 file and enter the password set during the extraction process.

The IP Office will advise you that the system and applications will stop responding while the certificate is applied. Click OK and wait for the IP Office to apply the certificate. Once applied you can view the certificate by clicking View.

Now that the IP Office is presenting the ipo.example.ca certificate to users we need to ensure that this resolves properly. We will still get an error if we try to browse to the IP Office by IP Address, since the certificate doesn't contain the internal IP. The internal DNS Server needs to point ipo.example.ca to the internal IP of the IP Office. As with many Avaya applications the IP Office performs a reverse DNS lookup to validate the FQDN. If you skip this step the IP Office will server an error page when you browse to it. Make sure that the IP Office is set to use the DNS Server that hosts your internal DNS entries. If you want your IP Office to be accessible to the Internet you can enable port forwarding on your router for ports 80, 443, 7070, 7071, and 8443. Otherwise, disable the port forwarding that was set up for the certificate creation. 

This process is a manual process that can't be automated at this time. You will need to update the certificate before it expires, following the entire process again. While this is not an ideal scenario, it is the easiest way to get free TLS certificates for your IP Office. The same process can be applied to a UC Module or Application Server - even Server Edition. Just follow the steps and remember that each element will require a unique, DNS-resolvable FQDN.

One final piece of housekeeping - the IP Office will, by default, generate an alarm when the certificate is within 60 days of expiring. To change this open the security settings, navigate to System, then Certificates. Change the Certificate Expiration Warning (days) to 30. This is the lowest number possible, so a System Status Alarm (or SNMP alarm if you've configured that) will be generated when the certificate is set to expire. IP Office Manager will also generate an alarm when logging in. I haven't found a way to eliminate this that doesn't do away with all the application security altogether. If you come up with something please let me know in the comments. I will update my post and give credit where it is due.