NetBackup 8.1 and above, master server loses connectivity to the media servers for no apparent reason

Problem

NetBackup 8.1 and above, master server loses connectivity to the media servers for no apparent reason.  The media servers can be any supported version of NetBackup, the ability to communicate to the master is possible (via ssh, telnet, ping), but when running NetBackup jobs or using the NetBackup commands to test connectivity will result in failures.

 

Error Message

Testing communications between servers using ' bptestbpcd '

<16>bptestbpcd main: Function ConnectToBPCD(exbackupaps1) failed: 8506
<16>bptestbpcd main: The certificate has expired.
<16>bptestbpcd main: Unable to perform peer host name validation. Curl error has occurred for peer name: <media_server>, self name: <master>: 8506
        [PROXY] Encountered error (VALIDATE_PEER_HOST_PROTOCOL_RUNNING) while processing(ValidatePeerHostProtocol).: 1
The certificate has expired.

 

Cause

In this case the Tomcat certificate has expired.  By design, the expiration date should auto-extend up to 180 days in advance of the expiration date.  In some cases the certificate does not auto-extended due to permission issues on certain key files or previous intervention where NetBackup secure Comms were rebuilt.

There are two methods to confirm that the Tomcat certificate has expired:
1 - With logging
Set the ' ENABLE_NBCURL_VERBOSE = 1 ' configuration perameter, then run ' nbcertcmd -ping ', and view the details in the ' nbcert ' log  (UNIX/Linux: /usr/openv/netbackup/logs/nbcert/   Windows: <installPath>\NetBackup\logs\nbcert\ ), and run the command ' nbcertcmd -listAllCertificates '.  The expiration of the Tomcat certificate *may* be different, but if either results show an expired certificate for Tomcat the certificate will need to be updated.


2 - With command line
NOTE: Adjust paths as necessary:
Windows:
"C:\Program Files\Veritas\NetBackup\jre\bin\keytool.exe" -list -v -keystore "C:\Program Files\Veritas\NetBackup\var\global\vxss\tomcatcreds\nbwebservice.jks" < "C:\Program Files\Veritas\NetBackup\var\global\jkskey" | findstr "until Owner"
"C:\Program Files\Veritas\NetBackup\jre\bin\keytool.exe" -list -v -keystore "C:\Program Files\Veritas\NetBackup\var\global\wsl\credentials\nbwebservice.jks" < "C:\Program Files\Veritas\NetBackup\var\global\jkskey" | findstr "until Owner"

UNIX/Linux:
/usr/openv/java/jre/bin/keytool -list -v -keystore /usr/openv/var/global/vxss/tomcatcreds/nbwebservice.jks < /usr/openv/var/global/jkskey | grep -e until -e Owner
/usr/openv/java/jre/bin/keytool -list -v -keystore /usr/openv/var/global/wsl/credentials/nbwebservice.jks < /usr/openv/var/global/jkskey | grep -e until -e Owner
 
The jkskey file contains the password needed to open the nbwebservice.jks file using the keytool command.  The command strings will open the file and feed it the password in one step. Each .jks file contains 3 certificates.  The middle Certificate is the TOMCAT certificate
 
Eaxmple Output:
Owner: O=vx, OU=root@jdhv2.community.veritas.com, CN=nbatd
Valid from: Tue Sep 04 11:46:55 CDT 2018 until: Mon Aug 30 13:01:55 CDT 2038
Owner: O=vx, OU=TOMCAT@jdhv2.community.veritas.com,CN=jdhv2      <- TOMCAT Identity String                          
Valid from: Tue Sep 04 11:47:59 CDT 2018 until: Wed Sep 04 13:02:59 CDT 2019            <- TOMCAT Cert Expiration Date
Owner: O=vx, OU=root@jdhv2.community.veritas.com, CN=broker
Valid from: Tue Sep 04 11:46:55 CDT 2018 until: Mon Aug 30 13:01:55 CDT 2038

In this case the TOMCAT certificate expires Wed Sep 04 13:02:59 CDT 2019. If the date displayed is prior to the current date than the certificate has expired and must be renewed.

 

Solution

The following commands will create a valid certificate

UNIX/Linux:

/usr/openv/netbackup/bin/admincmd/nbcertconfig - The -t and -f options will be needed for NetBackup versions 8.1.1 and higher
/usr/openv/wmc/bin/install/configureCerts
/usr/openv/wmc/bin/install/setupWmc
/usr/openv/netbackup/bin/nbwmc stop
/usr/openv/netbackup/bin/nbwmc start
/usr/openv/netbackup/bin/nbcertcmd -getCACertificate
/usr/openv/netbackup/bin/nbcertcmd -getCertificate -force

 

Windows:

On the Master, run services.msc and locate NetBackup Web Management Console service (nbwmc)
Identify the account used to start the nbwmc service
Locate /  Acquire the password for this account
Open an Administrator CMD prompt on the Master
Create the following Environment Variable for the CMD window by running:  set WEBSVC_PASSWORD=<passwordHere>
Run:  ...\NetBackup\bin\admincmd\nbcertconfig -t   (use   -t -f    for NBU 8.1.1 and newer
NOTE:  If this fails, it is likely to be due to an incorrect password.
To verify the password is correct, use the following command to spawn a new CMD window running as the account in question:  runas /user:<user> cmd.exe
Example local account : runas /user:nbwebsvc cmd.exe
Example domain account: runas /user:COMPANY\nbwebsvc cmd.exe
If the new CMD window opens successfully, it means the credentials were correct - the new window can simply be closed
If the new CMD window fails to open, examine the on-screen language to identify why
CD into ...\NetBackup\wmc\bin\install
Run: configureCerts.bat
Run: setupWmc
Restart the nbwmc service
Run:  nbcertcmd -getCACertificate
Run:  nbcertcmd -getCertificate –force

Note:  For clustered environments, please ensure to ' -cluster ' switch with ' nbcertcmd ' and/or refer to the commands reference guide for further details.

While it is preferred to have NetBackup auto-renew the certificates, the commands can also be run manually in advance of expiration. As the nbwmc process is cycled there may be a disruption to normal activities such as backups.