Source Code – Bi·lak

List all processes and how much swap they use

Sometimes I see that my computer uses a lot of swap space, but none of the system monitors (e.g. ps, Gnome’s System Monitor, top, btop) show which processes are to blame. There’re several memory metrics (e.g. total swap usage), but never per process swap usage. 😠

There’s a StackExchange question that asks the same thing, but the solutions are all not well scripted, slow or “meh” for other reasons … so I tackled the problem myself. 😝

All the information is basically buried in the /proc/*/status files. Among other things they contain the amount of swap a process uses, its PID and even better: its name. So we have to go through all the files, grep the useful lines, extract the data from those lines and recombine them.

I tried different combinations of grep and sed and Bash string interpolation … and while it worked, it was even slower than the StackExchange suggestions. 😯😅😞 This looked more and more like a “if I knew grep/sed/awk better I wouldn’t need to invoke 4 sub shells/pipes/processes for each file” kind of problem. I remembered Bryan Cantrill making an offhanded remark once that awk had a simple and concise language and a great manual.

If you get the awk programming language manual…you’ll read it in about two hours and then you’re done. That’s it. You know all of awk.
Bryan Cantrill

I had put off diving in for too long. So I started reading and roughly 20 minutes in I knew enough to solve the whole problem (almost) entirely in awk (it still needs the shell for filename globbing 😕🤷). And … it’s blazingly fast! And … you can also combine it with sort and head to quickly find the worst offenders. 😎

You can find the code in this Gist. There’s a simpler version in an earlier revision. 😉

Skip unreadable files in awk scripts

Have you ever needed to give awk multiple files to process, but wanted it to skip files that it can’t open? Use this snippet!

BEGINFILE {
    # skip files that cannot be opened
    if ( ERRNO != "" ) {
        nextfile
    }
}

BEGINFILE {

# skip files that cannot be opened

if ( ERRNO != "" ) {

nextfile

}

This trick is mentioned in the documentation for the BEGINFILE pattern, but not spelled out as code. You’re welcome. 😀

Moving Passwords From Firefox (Lockwise) to KeePassXC

Go to about:logins in Firefox.

Click on the three dots in the top right corner, select “Export Logins…” and save your passwords info a CSV file (e.g. exported_firefox_logins.csv).

There’s one complication: Firefox will save dates as Unix timestamps (with millisecond resolution) which KeePassXC doesn’t understand, so we’ll have to help it out.

Save the following script into a file (e.g. firefox_csv_date_fix.py).

#!/usr/bin/env python3
#
# Author: Riyad Preukschas &lt;riyad@informatik.uni-bremen.de&gt;
# License: Mozilla Public License 2.0
# SPDX-License-Identifier: MPL-2.0


import csv
import sys

from datetime import datetime


def main():
    if len(sys.argv) != 2:
        print("Usage: {} &lt;exported_firefox_logins.csv&gt;".format(sys.argv&#91;0]), file=sys.stderr)
        exit(1)

    csv_file_name = sys.argv&#91;1]

    with open(csv_file_name, 'rt') as f:
        # field names will be determined from first row
        csv_reader = csv.DictReader(f)
        # read all rows in one go
        rows = list(csv_reader)

    # add new columns with Iso-formatted dates
    for row in rows:
        row&#91;'timeCreatedIso'] = datetime.fromtimestamp(int(row&#91;'timeCreated'])/1000).isoformat()
        row&#91;'timeLastUsedIso'] = datetime.fromtimestamp(int(row&#91;'timeLastUsed'])/1000).isoformat()
        row&#91;'timePasswordChangedIso'] = datetime.fromtimestamp(int(row&#91;'timePasswordChanged'])/1000).isoformat()

    # write out the updated rows
    csv_writer = csv.DictWriter(sys.stdout, fieldnames=rows&#91;0].keys())
    csv_writer.writeheader()
    csv_writer.writerows(rows)


if __name__ == '__main__':
    main()

#!/usr/bin/env python3

# Author: Riyad Preukschas <riyad@informatik.uni-bremen.de>

# License: Mozilla Public License 2.0

# SPDX-License-Identifier: MPL-2.0

import csv

import sys

from datetime import datetime

def main():

if len(sys.argv) != 2:

print("Usage: {} <exported_firefox_logins.csv>".format(sys.argv[0]), file=sys.stderr)

exit(1)

csv_file_name = sys.argv[1]

with open(csv_file_name, 'rt') as f:

# field names will be determined from first row

csv_reader = csv.DictReader(f)

# read all rows in one go

rows = list(csv_reader)

# add new columns with Iso-formatted dates

for row in rows:

row['timeCreatedIso'] = datetime.fromtimestamp(int(row['timeCreated'])/1000).isoformat()

row['timeLastUsedIso'] = datetime.fromtimestamp(int(row['timeLastUsed'])/1000).isoformat()

row['timePasswordChangedIso'] = datetime.fromtimestamp(int(row['timePasswordChanged'])/1000).isoformat()

# write out the updated rows

csv_writer = csv.DictWriter(sys.stdout, fieldnames=rows[0].keys())

csv_writer.writeheader()

csv_writer.writerows(rows)

if __name__ == '__main__':

main()

Call it giving it the path of the exported_firefox_logins.csv file and redirect the output into a new file:

python3 ./firefox_csv_date_fix.py ./exported_firefox_logins.csv > ./fixed_firefox_logins.csv

It will add new columns (named: old column name + “Iso”) with the dates converted to the ISO 8601 format KeePassXC wants.

We can now use the fixed_firefox_logins.csv file to import the logins into KeePassXC.

Select Database -> Import -> CSV File... from the menu.

It will ask you to create a new database. Just do it you can get the data into your default database later (e.g. call it firefox_logins.kdbx for now).

In the import form select “First line has field names” and match up KeePassXC’s fields with the columns from the CSV:

KeePassXC Field	Firefox CSV Column	Notes
Group	Not Present
Title	url	Firefox doesn’t have a title field so we use the URL for it.
Username	username
Password	password
URL	url
Notes	httpRealm	I’m not sure how KeePassXC deals with HTTP authentication. There’s no special field for it, so I decided to keep it in the Notes field.
TOTP	Not Present
Icon	Not Present
Last Modified	timePasswordChangedIso	the “Iso” part is important!
Created	timeCreatedIso	the “Iso” part is important!

Have a look at the preview at the bottom. Does the data look sane? 🤨

Sadly KeePassXC doesn’t let us import Firefox’s timeLastUsedIso field into its Accessed field. 😑

All your imported Logins will be inside the “Root” group. I’d suggest creating a new group (e.g. “Firefox Import”) and moving all imported Logins into it.

You can now open your default database and use the Database -> Merge From Database ... menu item, select the firefox_logins.kdbx file to include it’s contents into the current database. You’ll see a new “Firefox Imports” group show up.

You can now move the single entries into the appropriate groups if you want (e.g. “KeePassXC-Browser Passwords” if you use the browser extension).

Scaffolding for fetching and parsing emails from IMAP with Python

A friend asked me if I could help him write a Python script for fetching and processing data from emails in his mailbox … Well, the thing with emails is that they’re a pain to work with (in any form). So, I tried to help him out with a little scaffolding (also available as a Gist).

App.нет

App.net is shutting down. ?

Exciting Unlimited Register Machines

A brief and entertaining talk by an obviously excited presenter. 🙂 It goes into the same directions as Jim Weirich’s talk about the Y combinator.

Aktivieren Sie JavaScript um das Video zu sehen.
https://www.youtube.com/watch?v=7Q-UwjgZ0q4

AR create_with

I knew there was AR’s find_or_create_by method but I didn’t know it had a very useful companion create_with .

@something ||= SomeModel.create_with(some: "stuff").find_or_create_by(foo: "bar")

1	@something \|\|= SomeModel.create_with(some: "stuff").find_or_create_by(foo: "bar")

This will either find a record SomeModel.where(foo: "bar") or create it with SomeModel.create(foo: "bar", some: "stuff") . Very useful.

AR Find Trickery

found = ModelA.find_by(id: id) || ModelB.find(id)

1	found = ModelA.find_by(id: id) \|\| ModelB.find(id)

I stumbled over this during a code review. It was supposed to look for a record in two distinct tables. find_by will return nil if nothing was found while find will raise . Hence either you get something (from any model) or you raise.

Backup And Restore Your Android Phone With ADB (And rsync)

Based on my previous scripts and inspired by two blog posts that I stumbled upon I tackled the “backup all my apps, settings and data” problem for my Android devices again. The “new” solutions both use rsync instead of adb pull for file transfers. They both use ADB to start a rsync daemon on the device, forward its ports to localhost and run rsync against it from your host.

Simon’s solution assumes your phone has rsync already (e.g. because you run CyanogenMod) and can become root via adb root . It clones all files from the phone (minus /dev , /sys , /proc etc.). He also configures udev to start the backup automatically when the phone is plugged in.

pts solves the setup without necessarily becoming root. He also has a way of providing a rsync binary to phones that don’t have any (e.g. when running OxygenOS). He also has a few tricks on how to debug the rsync daemon setup on the phone.

I’ve tried to combine both methods. My approach doesn’t require adb or rsync to be run as root. It’ll use the the system’s rsync when available or temporarily upload and use a backup one extracted from Cyanogen OS (for my OnePlus One). Android won’t allow you to chmod +x a file uploaded to /sdcard , but in /data/local/tmp it works. ?

The scripts will currently only backup and restore all of your /sdcard directory. Assuming you’re also using something like Titanium Backup you’ll be able to backup and restore all your apps, settings and data. To reduce the amount of data to copy it uses rsync filters to exclude caches and other files that you definitely don’t want synced ( .DS_Store files anyone?).

At the moment there’s one caveat: I had to disable restoring modification times (i.e. use --no-times ) because of an obnoxious error (they will be backuped fine, only restoring is the problem): ?

mkstemp “…” (in root) failed: Operation not permitted (1)

Additionally if you’re on the paranoid side you can also build your own rsync for Android to use as the backup binary.

The code and a ton of documentation can be found on GitHub. Comments and suggestions are welcome. ?

Build Rsync for Android Yourself

To build rsync for Android you’ll need to have the Android NDK installed already.

Then clone the rsync for android source (e.g. from ~~CyanogenMod~~ LineageOS) …

git clone https://github.com/LineageOS/android_external_rsync.git
cd android_external_rsync
# checkout the most recent branch
git checkout cm-14.1

git clone https://github.com/LineageOS/android_external_rsync.git

cd android_external_rsync

# checkout the most recent branch

git checkout cm-14.1

… create the missing
jni/Application.mk
build file (e.g. from this Gist) and adapt it to your case …

… and start the build with

export NDK_PROJECT_PATH=<code>pwd</code> ndk-build -d rsync

1	export NDK_PROJECT_PATH=<code>pwd</code> ndk-build -d rsync

You’ll find your self-build rsync in
obj/local/*/rsync
. ?

Update 2017-10-06:

Updated sources from CyanogenMod to LineageOS.
Added links to Gist and Andoid NDK docs
Updated steps to work with up-to-date setups

If you get something like the following warnings and errors …

[...]
./flist.c:454:16: warning: implicit declaration of function 'major' is invalid in C99
      [-Wimplicit-function-declaration]
                        if ((uint32)major(rdev) == rdev_major)
                                    ^
./flist.c:458:41: warning: implicit declaration of function 'minor' is invalid in C99
      [-Wimplicit-function-declaration]
                        if (protocol_version < 30 && (uint32)minor(rdev) <= 0xFFu)
                                                             ^
./flist.c:467:11: warning: implicit declaration of function 'makedev' is invalid in C99
      [-Wimplicit-function-declaration]
                        rdev = MAKEDEV(major(rdev), 0);
                               ^
./rsync.h:446:36: note: expanded from macro 'MAKEDEV'
#define MAKEDEV(devmajor,devminor) makedev(devmajor,devminor)
                                   ^
3 warnings generated.
[...]
./flist.c:473: error: undefined reference to 'makedev'
./flist.c:454: error: undefined reference to 'major'
./flist.c:457: error: undefined reference to 'major'
./flist.c:458: error: undefined reference to 'minor'
./flist.c:467: error: undefined reference to 'major'
./flist.c:467: error: undefined reference to 'makedev'
./flist.c:617: error: undefined reference to 'major'
./flist.c:619: error: undefined reference to 'minor'
./flist.c:621: error: undefined reference to 'minor'
./flist.c:788: error: undefined reference to 'makedev'
./flist.c:869: error: undefined reference to 'makedev'
./flist.c:1027: error: undefined reference to 'minor'
clang++: error: linker command failed with exit code 1 (use -v to see invocation) 
make: *** [obj/local/armeabi-v7a/rsync] Error 1

[...]

./flist.c:454:16: warning: implicit declaration of function 'major' is invalid in C99

[-Wimplicit-function-declaration]

if ((uint32)major(rdev) == rdev_major)

./flist.c:458:41: warning: implicit declaration of function 'minor' is invalid in C99

[-Wimplicit-function-declaration]

if (protocol_version < 30 && (uint32)minor(rdev) <= 0xFFu)

./flist.c:467:11: warning: implicit declaration of function 'makedev' is invalid in C99

[-Wimplicit-function-declaration]

rdev = MAKEDEV(major(rdev), 0);

./rsync.h:446:36: note: expanded from macro 'MAKEDEV'

#define MAKEDEV(devmajor,devminor) makedev(devmajor,devminor)

3 warnings generated.

[...]

./flist.c:473: error: undefined reference to 'makedev'

./flist.c:454: error: undefined reference to 'major'

./flist.c:457: error: undefined reference to 'major'

./flist.c:458: error: undefined reference to 'minor'

./flist.c:467: error: undefined reference to 'major'

./flist.c:467: error: undefined reference to 'makedev'

./flist.c:617: error: undefined reference to 'major'

./flist.c:619: error: undefined reference to 'minor'

./flist.c:621: error: undefined reference to 'minor'

./flist.c:788: error: undefined reference to 'makedev'

./flist.c:869: error: undefined reference to 'makedev'

./flist.c:1027: error: undefined reference to 'minor'

clang++: error: linker command failed with exit code 1 (use -v to see invocation)

make: *** [obj/local/armeabi-v7a/rsync] Error 1

… you probably need to update
config.h
and change
/* #undef MAJOR_IN_SYSMACROS */
to
#define MAJOR_IN_SYSMACROS 1
.