Bash – Bi·lak

List all processes and how much swap they use

Sometimes I see that my computer uses a lot of swap space, but none of the system monitors (e.g. ps, Gnome’s System Monitor, top, btop) show which processes are to blame. There’re several memory metrics (e.g. total swap usage), but never per process swap usage. 😠

There’s a StackExchange question that asks the same thing, but the solutions are all not well scripted, slow or “meh” for other reasons … so I tackled the problem myself. 😝

All the information is basically buried in the /proc/*/status files. Among other things they contain the amount of swap a process uses, its PID and even better: its name. So we have to go through all the files, grep the useful lines, extract the data from those lines and recombine them.

I tried different combinations of grep and sed and Bash string interpolation … and while it worked, it was even slower than the StackExchange suggestions. 😯😅😞 This looked more and more like a “if I knew grep/sed/awk better I wouldn’t need to invoke 4 sub shells/pipes/processes for each file” kind of problem. I remembered Bryan Cantrill making an offhanded remark once that awk had a simple and concise language and a great manual.

If you get the awk programming language manual…you’ll read it in about two hours and then you’re done. That’s it. You know all of awk.
Bryan Cantrill

I had put off diving in for too long. So I started reading and roughly 20 minutes in I knew enough to solve the whole problem (almost) entirely in awk (it still needs the shell for filename globbing 😕🤷). And … it’s blazingly fast! And … you can also combine it with sort and head to quickly find the worst offenders. 😎

You can find the code in this Gist. There’s a simpler version in an earlier revision. 😉

Skip unreadable files in awk scripts

Have you ever needed to give awk multiple files to process, but wanted it to skip files that it can’t open? Use this snippet!

BEGINFILE {
    # skip files that cannot be opened
    if ( ERRNO != "" ) {
        nextfile
    }
}

This trick is mentioned in the documentation for the BEGINFILE pattern, but not spelled out as code. You’re welcome. 😀

Backup And Restore Your Android Phone With ADB (And rsync)

Based on my previous scripts and inspired by two blog posts that I stumbled upon I tackled the “backup all my apps, settings and data” problem for my Android devices again. The “new” solutions both use

rsync

instead of

adb pull

for file transfers. They both use ADB to start a rsync daemon on the device, forward its ports to localhost and run rsync against it from your host.

Simon’s solution assumes your phone has rsync already (e.g. because you run CyanogenMod) and can become root via

adb root

. It clones all files from the phone (minus

/dev

/sys

/proc

etc.). He also configures udev to start the backup automatically when the phone is plugged in.

pts solves the setup without necessarily becoming root. He also has a way of providing a rsync binary to phones that don’t have any (e.g. when running OxygenOS). He also has a few tricks on how to debug the rsync daemon setup on the phone.

I’ve tried to combine both methods. My approach doesn’t require adb or rsync to be run as root. It’ll use the the system’s rsync when available or temporarily upload and use a backup one extracted from Cyanogen OS (for my OnePlus One). Android won’t allow you to

chmod +x

a file uploaded to

/sdcard

, but in

/data/local/tmp

it works. ?

The scripts will currently only backup and restore all of your

/sdcard

directory. Assuming you’re also using something like Titanium Backup you’ll be able to backup and restore all your apps, settings and data. To reduce the amount of data to copy it uses rsync filters to exclude caches and other files that you definitely don’t want synced (

.DS_Store

files anyone?).

At the moment there’s one caveat: I had to disable restoring modification times (i.e. use

--no-times

) because of an obnoxious error (they will be backuped fine, only restoring is the problem): ?

mkstemp “…” (in root) failed: Operation not permitted (1)

Additionally if you’re on the paranoid side you can also build your own rsync for Android to use as the backup binary.

The code and a ton of documentation can be found on GitHub. Comments and suggestions are welcome. ?