NSO - Troubleshooting, Backup &
Restore
Imtiaz Ahmad & Imran Baig
Platform Software Group / Cisco Advanced Services
June 2017
Agenda
§ NSO Logs & Troubleshooting Tips
§ NSO Logs
§ NSO Error Log commands
§ Changing Log Level
§ Log Rotation
§ Performance Impact by each Log file
§ NSO Backup & Restore
§ Backup Functionality
§ Restore Functionality
§ Alternate of ncs-backup tool
NSO Logs
• Log File Setup & Location
• Logging level can be set for most NSO logs via ncs.conf
• Located at /var/log/ncs (for system installs)
• Log Files
• audit.log – NB Calls including WebUI, JSON, REST API calls coming into NSO
• ncs.log – Logs NSO system events including issues with startup/shutdown
• ncs-java-vm.log – NSO’s Java VM issues
• devel.log – capture detailed info on Java code / CBD / sync-from and other details
NSO Logs - Continued
• Log Files
• ned-alu-..trace – generated by setting device trace on via NSO CLI (raw vs. pretty)
• xpath-trace-log – troubleshooting Xpath in YANG must statements (use carefully)
• ncserr.log – NSO Daemon errors are logged
• snmp.log – SNMP traps info
• netconf.log – Troubleshooting NB NETCONF operations coming into NSO
• ncs-python-vm.log – NSO python VM issues
• ncs-python-vm-<package-name>.log – Troubleshooting Python package specific issues
NSO Logs - Commands
• Commands
• ncs –debug-dump myncsdebug : send this to NSO Dev Team or attach to case
• ncs --printlog ncserr.log.1 :send this to NSO Dev Team
• ncs_cmd –c reopen_logs : restarting Logging without restarting NSO
Changing Log level
• Changing logging level for ’devel.log’ in >> ncs.conf
• Changing device trace setting >> NSO CLI
Log Rotation
§ The primary agent for log rotation is the Linux logrotate utility.
§ All files under /var/log/ncs are subject to rotation by logrotate, except ncserr.log*
§ Rotation ncserr.log* files are managed by NCS itself.
§ Rotation Options
§ createolddir :Create ARCHIVED dir if it doesn't exit.
§ compress :Compress rotated log files.
§ copytruncate :Truncate file during rotation.
§ dateext :Use a date as the arvhiced file extension.
§ dateformat :Set the file extension format to be -YYYYmmdd
§ maxage 90 :Delete archived log files older than 90 days
§ maxsize 15728640 :Rotate any log file >= to 15MB
§ missingok :Do not issue an error if a log file is missing
§ ifempty :Rotate the log file even if it’s empty
§ maxage :Maximum retention days of a rotated log file.
§ nosharedscripts :The "prerotate" script applies target individually.
§ olddir :Location of rotated files.
Log File Information System Installation Local Install Impact
Enabled by Default Enabled by Default
ncs-python- Python VM Log Yes Yes Low Impact
vm.log
ncs-python-vm- Python Yes Yes Medium
<package- Package Log Impact
name>.log
ned-alu-..trace NED Trace Log No No High Impact
NSO Backup & Restore
Backup / Restore CDB Backup
• Simple CDB Backup/Restore Option
• Backup entire CDB or specific service
• ncs_load nso-config-backup.xml
• ncs_load –P /ncs:services/l3mplsvpn:l3mplsvpn nso-l3mplsvpn.xml
• Restore CDB or specific Service
• ncs_load –u admin –j –1 nso-config-backup.xml
• ncs_load –u admin –j –1 –m nso-l3mplsvpn.xml
• Use -m (merge) option instead of outright replacing configuration
Backup / Restore
• Backup
• Built-in functionality
• Available with ‘system-install’
• Backs-up following files/components
• CDB
• State files
• Rollback files
• Configuration files
Backup / Restore
• Backup Process
• No need to Stop NSO
• Run “ncs-backup” command
• Stored at /var/opt/ncs/backups/ folder
• Run it periodically using CRON job
• For disaster recovery, recommended to copy backup to remote server
Backup / Restore
• Restore
• Restore full running environment
• Entire configuration replaced
• Latest configuration changes may be lost
• Time consuming for very large configuration
• Restore Process
• Stop NSO
• ncs-backup –restore <path-to-backup-file>
• Start NSO
Backup / Restore
• Complex Backup
• NSO in HA
• NSO with Clustering & HA
• NSO with LSA & HA
• Solution
• nct tool - useful for multiple NSO nodes
• Backup & Restore
• Upgrade
• Install & Uninstall
• …..
