Monitor Progress with Pipe Viewer

One of the most frustrating things when doing system administration is having no idea how long a process will take to finish or how much progress it’s made. To get a better look at what’s going on, try the Pipe Viewer utility.

For sys admins, running a process can feel like driving cross-country without a road map. Pipe Viewer is one way to answer the question, ‘are we there yet?’

Pipe Viewer, or just pv when you’re invoking it at the command line or in scripts, is a utility for monitoring data flowing through a pipeline. It gives you an idea of how fast the data is moving through the pipeline, how long it’s taken so far, and when it will be finished. It’s the digital answer to the administrative question, “are we there yet?”

Pipe Viewer is not installed by default on most distros I’ve seen, so you’ll need to look for packages on some systems. It’s packaged for Debian and Ubuntu, so you can grab pv with the standard apt-get install pv dance.

We’ll start with a really simple example of pv in action. Let’s say you want to see how long it’s going to take to compress a logfile with gzip. Run pv logfile | gzip > logfile.gz. A progress bar will demonstrate the amount of progress made and how much further it must go. If you’re doing this with a small file, pv is going to add little to the process — it will be over before you can blink.

Let me be the first to say (at least in this post …) that pv can be “some assembly required.” That is, it can really report accurately only if it knows what it’s reporting — and it relies on the user to tell it, in some cases. This can be easily overcome, however, with a little extra help from du and cut. Hat tip to the Super User Stack Exchange for this one.

If you want to see the progress of, say, creating a compressed tarball from a directory, you can grab the size and then pass it to pv. First, get the size of the directory using SIZE=`du -sk directory | cut -f 1`. This will use du to grab the size of the directory and then pass it to cut to grab the proper field, saving it as SIZE.

Next, run the job: tar cf - directory | pv -p -s ${SIZE}k | bzip2 -c > directory.tar.bz2. This will tar up the directory while passing it to pv, which will then pass the output to bzip2. Since it knows the size, pv can report the progress (-p) while it’s going on. Note that the Stack Exchange example also shows using the v (verbose) option with tar. You don’t want to do that, as it will interfere with the output you want to see from pv.

Really, pv should be part of any admin’s toolbox. While some utilities are advanced enough that they have their own built-in progress reporting, many (like tar) leave that as an exercise to the user. And when it comes to jobs that require using several utilities and passing data through pipes, pv is just what the admin ordered.

Joe ‘Zonker’
Brockmeier is a freelance writer and editor with more than 10 years covering IT. Formerly the openSUSE Community Manager for Novell, Brockmeier has written for Linux Magazine, Sys Admin, Linux Pro Magazine, IBM developerWorks, Linux.com, CIO.com, Linux Weekly News, ZDNet, and many other publications. You can reach Zonker at jzb@zonker.net and follow him on Twitter.

Follow ServerWatch on Twitter