// I need to

ModernizeUpgrade without breaking it RescueFix what someone else started Build NewTurn an idea into a product Reach More PeopleGet found, get used, include everyone

// Company

Case StudiesExpertiseAbout + CareersResourcesContact Book a call →
≡articleBuild New

Project Plumbing with Plumbum (Part II)

Plumbum part 2: run and manage shell commands on remote machines over SSH from Python using SshMachine, including as a context manager.

Art+Logic Apr 24, 20144 min read

From the Art+Logic archive · written in 2014. The tools may have moved on; the thinking still holds.

Image of 3D pipes
image via http://www.berfrois.com/2011/12/piped/

So if you read my previous article, you’ve hopefully found that Plumbum is a fantastic tool for running and managing shell tools from Python. But stay tuned, dear readers, because we’ve only scratched the surface of the fantastic facilities that plumbum provides.

Spooky Plumbing at a Distance

When last we spoke we talked a lot about the LocalMachine object, the name of which implies that there is also some other kind of some other, non-local kind of machine. And Plumbum in fact does provide such a thing. In fact it provides several, all based around the idea of running commands on a remote machine over the SSH protocol.

Let’s start with the first implemented and most well supported, the SshMachine, which you create like this:

from plumbum import SshMachine
remote = SshMachine('hostname', user='username', keyfile='/path/to/keyfile')
--commands go here--
remote.close()

If you prefer, you can also use an SshMachine as a context manager:

with SshMachine('hostname', user='username', keyfile='/path/to/keyfile') as remote:
   # commands go here
   pass

Once you have a reference to the remote machine, you can use it pretty almost exactly like the LocalMachine we discussed in the last article. This means that you can do things like create a Python function that takes a machine as an argument, and (with a little care and a few caveats) it will work just the same whether you give it a local or remote machine object. Throw some plumbum shell commands in that function, and maybe iterate over a list of hostnames, and you have the makings of a basic host management system. Something like this:

def fortuna(machine):
   fcmd = machine['fortune']
   return fcmd()
hosts = ['localhost', 'host1', host2', 'host3]
username = 'username'
for host in hosts:
   if host == 'localhost':
      from plumbum import local as m
   else:
      m = SshMachine(host, user=username, keyfile=kfile)
   print host, fortuna(m)

See? Fun! Obviously, you’d probably want to use something a bit less whimsical than the fortune command in your own work. Just as obviously, you’d probably want to include some error handling or something.

Tunnelling - But Not the Quantum Kind

In addition to providing analogues of the same facilities that LocalMachine gives you, such as cwd and env attributes, SshMachine also provides access to the tunnelling functions of the SSH protocol. Just use the tunnel method, like so:

rem = SshMachine(host, user=username, keyfile=kfile)
tunl = rem.tunnel(local_port, remote_port)
#do stuff here
tunl.close()

Or, as a context manager:

rem = SshMachine(host, user=username, keyfile=kfile)
with rem.tunnel(local_port, remote_port) as tunl:
   # do stuff here

Once you have a tunnel set up, you can connect a regular Python socket to the local port, and traffic will be transparently passed across the tunnel to the remote port.

Beware the Pipes!

One thing to watch out for when using Plumbum to manage remote commands, is that piping commands together may not work the way you expect. Although this should be fixed in a future release, as of this writing, all pipes get connected on the local machine. This means that if you pipe together two remote commands using the | operator, the output of the first command will get sent back to the local machine, then sent to the remote machine as input to the second command, and finally the output of the second command sent back to the local machine. If you have a lot of data being piped around, this could be really, really slow. So be aware!

Once more, with Windows!

Since SshMachine wraps the ssh and scp executables for most of its functionality, and most Windows machines won’t have these, you might be thinking that the above isn’t very helpful for Windows users. Don’t fret, true believers, Plumbum has you covered. Remember near the beginning of this article, when I said Plumbum provides several types of remote machine objects? Well, in addition to the SshMachine object, there’s also a PuttyMachine, which works in basically the same way, but it wraps the plink and pscp executables included with PuTTY, which is a common SSH client for Windows.

The Final Option

The third and final type of remote machine object included with Plumbum is the ParamikoMachine, which wraps a pure-Python SSH implementation library called, unsurprisingly, Paramiko.

This one works a bit differently, since it’s wrapping a library, rather than external executables. For example, Plumbum’s piping and command redirection (which are based on using OS-level file descriptors) don’t really work. On the other hand, Plumbum gets a lot more control over the underlying SSH connection, so you can do things like creating multiple SSH tunnels over a single SSH connection (though the tunneling API is different). So the ParamikoMachine might be a better choice for some use cases, but isn’t really interchangeable with the SshMachine, so your mileage, as they say, may vary.

Paths and the utils module

Just like LocalMachine objects can generate LocalPath objects, the various types of RemoteMachine can generate RemotePath objects. These are kind of nice but aren’t really impressive until you take a look at the plumbum.utils module, and it’s three utility functions, copy, move, and delete. These do pretty much what you’d expect, but the spiffy part is that they take Plumbum path objects as arguments, and will automatically do the right thing, based on whether those paths are local or remote. So you can manipulate files and directories across machines in only a couple of lines of code:

with SshMachine(host, user=username, keyfile=kfile) as rem:
   copy(local.path('/some/path/here'), rem.path('/some/other/path/here'))

Even better, you can write reusable code which takes either local or remote paths as arguments.

But wait, there’s more!

Yes, it’s true, even after two longish posts about all the stuff you can do with Plumbum, we’re still not done yet. There’s a whole other part of Plumbum, which we haven’t even touched yet… writing command line applications using the included application framework!

Stay tuned!

keep going

More to explore in Build New.

All Build New resources →
let's talk

Got a problem like this one?

Tell us what's stuck. A real person replies within one business day.

Talk to our team →

Talk to people who know software, not a sales script.

Your first call is with Aja, our VP of Sales. He'll help make sure you're solving the right problem. Here's what you'll cover together:

  1. What you're trying to accomplish

    The business goal first, then the technology.

  2. Possible approaches

    And the tradeoffs of each, in plain language.

  3. Risks you may not have considered

    Technical and business, before they cost you.

  4. Whether AI changes the math

    What's practical, and what's just hype for your project.

  5. Sensible next steps

    Even if the right next step is not us.

book-a-call.ics

Loading the scheduler…

Open the booking page →

Some problems are easier to explain in writing. Send us a few lines about what you're working on, and we'll reply within one business day. Helpful to include:

  1. What the software does, or should do
  2. What's stuck, or what's worrying you
  3. Any deadline or budget you have in mind
note.md

Loading the form…