​How to identify and correct certificate and CRL related errors in 8.1 and later. Status codes/errors: hung jobs, offline media servers, 13, 7625, 7660

How to identify and correct certificate and CRL related errors in 8.1Article: 100039941
Last Published: 2017-09-15
Ratings:  9 1
Product(s): NetBackupProblemCorruption of files can occur due to unexpected system events, such as the sudden shutdown or reboot of the system, hardware errors, such as a faulty disks, or software/operating system type errors triggering incorrect writes of data to disk.  This article will explain how to identify and correct certificate or Certificate Revocation List (CRL) related issues in 8.1 to include the following issues:
  A corrupted certmapinfo.json file

Resulting in status codes/errors include:  13, 5949, 7660

 A corrupted certificate authority (CA) certificate

Status codes/errors:  hung jobs, offline media servers, 13, 7625, 7660

 A corrupted local host ID-based certificate.

Status codes/errors: 25, 7625, 5942

 A corrupt certificate revocation list (CRL) 

Status codes/errors: 23, 25, 7640, 7654, 9301 
 

 Error MessageError codes, status codes and messages that may be experienced include, but are not limited to the following:
 Error Message
13 File read failed
23 Socket read failed.
25 Cannot connect on socket.
61 The vnetd proxy encountered an error
5949 Certificate does not exist.
5978 Unable to read the certificate mapping file.
5942 Certificate could not be read from the local certificate store
7624 SSL accept failed.
7625 SSL socket connect failed
7627 PEM_X509_INFO_read_bio failed
7640 The peer closed the connection
7660 The peer host certificate cannot be verified using the Certificate Revocation List.
9301 Failed to decode certificate revocation list
N/A Backups hung waiting for resources. 
 CauseBest Practices
Certificate, CRL and certmapinfo.json file problems are easiest to identify on the client, or media server reporting the issue. 

 
 
Log files that should be enabled on the server reporting the issue, when troubleshooting the error codes listed above when they are related to certificate or CRL type issues are:
 Log file directory VERBOSE or Debug Level OID
*nbpxyhelper DebugLevel=4 486
nbcert VERBOSE = 5  
bpcd VERBOSE = 5  
bprd (master server only) VERBOSE = 5   
* Unix/Linux operating systems, this is located in /usr/openv/logs.  All other log directories referenced are located in netbackup/logs for both Unix/Linux and Windows.
 
**Note that the DiagnosticLevel should always be set to 6.
 
 
The commands, nbcertcmd, bptestbpcd and bpclntcmd, are useful in troubleshooting certificate and CRL related issues.  Common syntax of the commands utilized in troubleshooting are:
sudo /usr/openv/netbackup/bin/bpclntcmd -pn
sudo /usr/openv/netbackup/bin/bpclntcmd -hn <hostname>
sudo /usr/openv/netbackup/bin/bptestbpcd -host [host]
sudo /usr/openv/netbackup/bin/nbcertcmd -getCRL
sudo /usr/openv/netbackup/bin/nbcertcmd -getCertificate
sudo /usr/openv/netbackup/bin/nbcertcmd -getCertificate -force
sudo /usr/openv/netbackup/bin/nbcertcmd -getCertificate -token <reissue_token> -force
sudo /usr/openv/netbackup/bin/nbcertcmd -getCACertificate
sudo /usr/openv/netbackup/bin/nbcertcmd -hostselfcheck
sudo /usr/openv/netbackup/bin/nbcertcmd -listCACertDetails
sudo /usr/openv/netbackup/bin/nbcertcmd -listCertDetails
 
The commands are all located in:
Windows:  <install_path>\Veritas\NetBackup\bin
 Unix/Linux:  /usr/openv/netbackup/bin
 
For additional information, see the NetBackup 8.1 Commands Reference Guide.
 
 Solution
Select the following links to be directed identification and solution of each issue:
Issue 1:  A corrupted certmapinfo.json file. 
Issue 2: A corrupted certificate authority (CA) certificate
Issue 3: A corrupted local host ID-based certificate 
Issue 4 : A corrupt certificate revocation list (CRL)