In the lockdown period, during pandemic, there is nothing much to do over weekend 😏.
Was just thinking of writing ✍️ something, something useful…. then came up with an idea to address a common problem that many sysadmins and netadmins come across.
At times, we want to run set of commands on many devices, hosts to perform some bulk changes, like run package updates, or get some information, like disk space, from set of servers.
To achieve this, I have designed a simple json schema which has predefined credential sets (along with sudo/enable password option), command alias and hosts mapped with credential set and list of command aliases.
Also, I have added multiprocessing in code to run commands concurrently on “maxhosts” defined in json file.
Here is config.json which defined all above stuff, pretty simple to understand.
config.json:
{
"maxhosts": 5,
"credsets": {
"set1": { "username": "manish", "password": "abcd", "enpass": "xyz123" },
"set2": { "username": "pi", "password": "Hello!" }
},
"cmds": {
"listfiles": ["ls -l /home", 2],
"diskspace": ["df -h", 1],
"uptime": ["uptime", 1],
"updatedebian": ["apt-get update && apt-get upgrade -y", 10],
"DebVer": ["cat /etc/debian_version", 1]
},
"hosts": {
"192.168.0.6": {"type": "linux", "credset": "set1", "cmds": ["updatedebian", "DebVer"] },
"192.168.0.7": {"type": "linux", "credset": "set1", "port": 1223, "cmds": ["updatedebian"] }
},
"logging": {
"logfile": "/home/manish/pymgt/test.logs"
}
}Above config.json is picked by following python code to first extract list of servers/hosts. Then get associated credential and command list to be executed. Each command has delay factor in json (see netmiko documentation), default delay is about 100 sec, so if a command has delay factor of 3, then delay will be close to 300 sec. This is useful when you are performing upgrades etc. Output of login activity, command executions can be seen on standard output and later in logfile (defined in config.json).
pymgt.py:
#!/usr/bin/python3
from netmiko import ConnectHandler
from multiprocessing import Pool
import os
import json
from time import strftime, localtime
import sys
try:
params = sys.argv[1]
param, value = params.split('=')
if param != "--play":
sys.exit()
playconf = value
except:
print("Usage: pymgt.py --play=<json config>")
sys.exit()
with open(playconf) as cfg:
cfgdata = json.load(cfg)
def CommitLogs(LogMessage):
logfile = cfgdata.get('logging').get('logfile')
try:
fopen = open(logfile, "a")
try:
fopen.write(LogMessage+"\n")
fopen.close()
except:
print("Failed to write ",LogMessage)
return
except:
print("failed to open file", logfile)
return
def injectcmds(device, cmds):
try:
net_connect = ConnectHandler(**device)
try:
LogMessage = strftime("%Y-%m-%d %H:%M:%S", localtime())+" "+device.get('host')+" Logged in..."
print(LogMessage)
CommitLogs(LogMessage)
if device.get('secret') is not None:
net_connect.enable()
LogMessage = strftime("%Y-%m-%d %H:%M:%S", localtime())+" "+device.get('host')+" Enabled Elevated access...."
print(LogMessage)
CommitLogs(LogMessage)
sshstatus = 0
except:
LogMessage = strftime("%Y-%m-%d %H:%M:%S", localtime())+" "+device.get('host')+";Cannot gain elevated access...."
print(LogMessage)
CommitLogs(LogMessage)
sshstatus = -1
except:
LogMessage = strftime("%Y-%m-%d %H:%M:%S", localtime())+" "+device.get('host')+";SSH failed as "+device.get('username')
print(LogMessage)
CommitLogs(LogMessage)
sshstatus = -1
if (sshstatus == 0):
for cmd in cmds:
waittime = 10
cmdobj = cfgdata.get('cmds').get(cmd)
cmd, delayfac = cmdobj[0],cmdobj[1]
print(cmd, delayfac)
try:
status = net_connect.send_command(cmd, delay_factor=delayfac)
LogMessage = strftime("%Y-%m-%d %H:%M:%S", localtime())+" "+device.get('host')+" Command:"+cmd+"\nOutput:"+status
print(LogMessage)
CommitLogs(LogMessage)
except:
LogMessage = strftime("%Y-%m-%d %H:%M:%S", localtime())+" "+device.get('host')+" "+cmd+" command failed"
print(LogMessage)
CommitLogs(LogMessage)
return
def hostcmd(host):
hostdata = cfgdata.get('hosts').get(host)
type = hostdata.get('type')
cmds = hostdata.get('cmds')
port = hostdata.get('port')
if port is None:
port = 22
credset = hostdata.get('credset')
credparams = cfgdata.get('credsets').get(credset).keys()
username = cfgdata.get('credsets').get(credset).get("username")
password = cfgdata.get('credsets').get(credset).get("password")
if "enpass" in credparams:
enpass = cfgdata.get('credsets').get(credset).get("enpass")
else:
enpass = None
# print("Enpass", host, enpass)
if enpass is None:
device = {
'device_type': type,
'host': host,
'port': port,
'username': username,
'password': password,
}
else:
device = {
'device_type': type,
'host': host,
'port': port,
'username': username,
'password': password,
'secret': enpass,
}
injectcmds(device, cmds)
hostlist = list(cfgdata.get('hosts').keys())
maxhosts = cfgdata.get('maxhosts')
with Pool(maxhosts) as p:
p.map(hostcmd, hostlist)
Lets execute it
manish@hq:~/pymgt $ ./pymgt.py --play=/home/manish/pymgt/config.json
2020-04-12 08:40:56 192.168.0.6 Logged in...
2020-04-12 08:40:57 cloud.mka.in Logged in...
2020-04-12 08:40:58 192.168.0.6 Enabled Elevated access....
apt-get update && apt-get upgrade -y 10
2020-04-12 08:40:59 cloud.mka.in Enabled Elevated access....
apt-get update && apt-get upgrade -y 10
........
........Here is the sample log output saved in test.log (defined in config.json)
cat test.logs
2020-04-04 18:39:03 192.168.0.7 Logged in...
2020-04-04 18:39:03 192.168.0.6 Logged in...
2020-04-04 18:39:04 192.168.0.7 Command:df -h
Output:Filesystem Size Used Avail Use% Mounted on
/dev/root 7.1G 3.6G 3.3G 52% /
devtmpfs 459M 0 459M 0% /dev
tmpfs 464M 115M 349M 25% /dev/shm
tmpfs 464M 6.4M 457M 2% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 464M 0 464M 0% /sys/fs/cgroup
/dev/mmcblk0p1 253M 53M 200M 21% /boot
tmpfs 93M 0 93M 0% /run/user/1000
tmpfs 93M 0 93M 0% /run/user/999
tmpfs 93M 0 93M 0% /run/user/1001
2020-04-04 18:39:04 192.168.0.6 Enabled Elevated access....
2020-04-04 18:39:05 192.168.0.6 Command:ls -l /home
Output:total 8
drwxr-xr-x 7 manish manish 4096 Apr 4 16:17 manish
drwxr-xr-x 17 pi pi 4096 Apr 4 15:29 pi
2020-04-04 18:39:06 192.168.0.6 Command:uptime
Output: 18:39:06 up 2:27, 6 users, load average: 2.65, 2.56, 2.65Hope readers it useful :-), specially sysadms and netadms.