Project Plumbing with Plumbum (Part II)


Image of 3D pipes

image via

So if you read my previous article, you’ve hopefully found that Plumbum is a fantastic tool for running and managing shell tools from Python. But stay tuned, dear readers, because we’ve only scratched the surface of the fantastic facilities that plumbum provides.

Spooky Plumbing at a Distance

When last we spoke we talked a lot about the LocalMachine object, the name of which implies that there is also some other kind of some other, non-local kind of machine. And Plumbum in fact does provide such a thing. In fact it provides several, all based around the idea of running commands on a remote machine over the SSH protocol.

Let’s start with the first implemented and most well supported, the SshMachine, which you create like this:

from plumbum import SshMachine
remote = SshMachine('hostname', user='username', keyfile='/path/to/keyfile')
--commands go here--

If you prefer, you can also use an SshMachine as a context manager:

with SshMachine('hostname', user='username', keyfile='/path/to/keyfile') as remote:
   # commands go here

Once you have a reference to the remote machine, you can use it pretty almost exactly like the LocalMachine we discussed in the last article. This means that you can do things like create a Python function that takes a machine as an argument, and (with a little care and a few caveats) it will work just the same whether you give it a local or remote machine object. Throw some plumbum shell commands in that function, and maybe iterate over a list of hostnames, and you have the makings of a basic host management system. Something like this:

def fortuna(machine):
   fcmd = machine['fortune']
   return fcmd()
hosts = ['localhost', 'host1', host2', 'host3]
username = 'username'
for host in hosts:
   if host == 'localhost':
      from plumbum import local as m
      m = SshMachine(host, user=username, keyfile=kfile)
   print host, fortuna(m)

See? Fun! Obviously, you’d probably want to use something a bit less whimsical than the fortune command in your own work. Just as obviously, you’d probably want to include some error handling or something.

Tunnelling – But Not the Quantum Kind

In addition to providing analogues of the same facilities that LocalMachine gives you, such as cwd and env attributes, SshMachine also provides access to the tunnelling functions of the SSH protocol. Just use the tunnel method, like so:

rem = SshMachine(host, user=username, keyfile=kfile)
tunl = rem.tunnel(local_port, remote_port)
#do stuff here

Or, as a context manager:

rem = SshMachine(host, user=username, keyfile=kfile)
with rem.tunnel(local_port, remote_port) as tunl:
   # do stuff here

Once you have a tunnel set up, you can connect a regular Python socket to the local port, and traffic will be transparently passed across the tunnel to the remote port.

Beware the Pipes!

One thing to watch out for when using Plumbum to manage remote commands, is that piping commands together may not work the way you expect. Although this should be fixed in a future release, as of this writing, all pipes get connected on the local machine. This means that if you pipe together two remote commands using the | operator, the output of the first command will get sent back to the local machine, then sent to the remote machine as input to the second command, and finally the output of the second command sent back to the local machine. If you have a lot of data being piped around, this could be really, really slow. So be aware!

Once more, with Windows!

Since SshMachine wraps the ssh and scp executables for most of its functionality, and most Windows machines won’t have these, you might be thinking that the above isn’t very helpful for Windows users. Don’t fret, true believers, Plumbum has you covered. Remember near the beginning of this article, when I said Plumbum provides several types of remote machine objects? Well, in addition to the SshMachine object, there’s also a PuttyMachine, which works in basically the same way, but it wraps the plink and pscp executables included with PuTTY, which is a common SSH client for Windows.

The Final Option

The third and final type of remote machine object included with Plumbum is the ParamikoMachine, which wraps a pure-Python SSH implementation library called, unsurprisingly, Paramiko.

This one works a bit differently, since it’s wrapping a library, rather than external executables. For example, Plumbum’s piping and command redirection (which are based on using OS-level file descriptors) don’t really work. On the other hand, Plumbum gets a lot more control over the underlying SSH connection, so you can do things like creating multiple SSH tunnels over a single SSH connection (though the tunneling API is different). So the ParamikoMachine might be a better choice for some use cases, but isn’t really interchangeable with the SshMachine, so your mileage, as they say, may vary.

Paths and the utils module

Just like LocalMachine objects can generate LocalPath objects, the various types of RemoteMachine can generate RemotePath objects. These are kind of nice but aren’t really impressive until you take a look at the plumbum.utils module, and it’s three utility functions, copy, move, and delete. These do pretty much what you’d expect, but the spiffy part is that they take Plumbum path objects as arguments, and will automatically do the right thing, based on whether those paths are local or remote. So you can manipulate files and directories across machines in only a couple of lines of code:

with SshMachine(host, user=username, keyfile=kfile) as rem:
   copy(local.path('/some/path/here'), rem.path('/some/other/path/here'))

Even better, you can write reusable code which takes either local or remote paths as arguments.

But wait, there’s more!

Yes, it’s true, even after two longish posts about all the stuff you can do with Plumbum, we’re still not done yet. There’s a whole other part of Plumbum, which we haven’t even touched yet… writing command line applications using the included application framework!

Stay tuned!

+ more

Accurate Timing

Accurate Timing

In many tasks we need to do something at given intervals of time. The most obvious ways may not give you the best results. Time? Meh. The most basic tasks that don't have what you might call CPU-scale time requirements can be handled with the usual language and...

read more