A friend asked me if I could help him write a Python script for fetching and processing data from emails in his mailbox … Well, the thing with emails is that they’re a pain to work with (in any form). So, I tried to help him out with a little scaffolding (also available as a Gist).
@something ||= SomeModel.create_with(some: "stuff").find_or_create_by(foo: "bar")
This will either find a record SomeModel.where(foo: "bar") or create it with SomeModel.create(foo: "bar", some: "stuff") . Very useful.
found = ModelA.find_by(id: id) || ModelB.find(id)
I stumbled over this during a code review. It was supposed to look for a record in two distinct tables. find_by will return nil if nothing was found while find will raise . Hence either you get something (from any model) or you raise.
Based on my previous scripts and inspired by two blog posts that I stumbled upon I tackled the “backup all my apps, settings and data” problem for my Android devices again. The “new” solutions both use rsync instead of adb pull for file transfers. They both use ADB to start a rsync daemon on the device, forward its ports to localhost and run rsync against it from your host.
Simon’s solution assumes your phone has rsync already (e.g. because you run CyanogenMod) and can become root via adb root . It clones all files from the phone (minus /dev , /sys , /proc etc.). He also configures udev to start the backup automatically when the phone is plugged in.
pts solves the setup without necessarily becoming root. He also has a way of providing a rsync binary to phones that don’t have any (e.g. when running OxygenOS). He also has a few tricks on how to debug the rsync daemon setup on the phone.
I’ve tried to combine both methods. My approach doesn’t require adb or rsync to be run as root. It’ll use the the system’s rsync when available or temporarily upload and use a backup one extracted from Cyanogen OS (for my OnePlus One). Android won’t allow you to chmod +x a file uploaded to /sdcard , but in /data/local/tmp it works. ?
The scripts will currently only backup and restore all of your /sdcard directory. Assuming you’re also using something like Titanium Backup you’ll be able to backup and restore all your apps, settings and data. To reduce the amount of data to copy it uses rsync filters to exclude caches and other files that you definitely don’t want synced ( .DS_Store files anyone?).
At the moment there’s one caveat: I had to disable restoring modification times (i.e. use --no-times ) because of an obnoxious error (they will be backuped fine, only restoring is the problem): ?
mkstemp “…” (in root) failed: Operation not permitted (1)
Additionally if you’re on the paranoid side you can also build your own rsync for Android to use as the backup binary.
The code and a ton of documentation can be found on GitHub. Comments and suggestions are welcome. ?
To build rsync for Android you’ll need to have the Android NDK installed already.
git clone https://github.com/LineageOS/android_external_rsync.git
# checkout the most recent branch
git checkout cm-14.1
… and start the build with
export NDK_PROJECT_PATH=<code>pwd</code> ndk-build -d rsync
You’ll find your self-build rsync in
- Updated sources from CyanogenMod to LineageOS.
- Added links to Gist and Andoid NDK docs
- Updated steps to work with up-to-date setups
If you get something like the following warnings and errors …
./flist.c:454:16: warning: implicit declaration of function 'major' is invalid in C99
if ((uint32)major(rdev) == rdev_major)
./flist.c:458:41: warning: implicit declaration of function 'minor' is invalid in C99
if (protocol_version < 30 && (uint32)minor(rdev) <= 0xFFu)
./flist.c:467:11: warning: implicit declaration of function 'makedev' is invalid in C99
rdev = MAKEDEV(major(rdev), 0);
./rsync.h:446:36: note: expanded from macro 'MAKEDEV'
#define MAKEDEV(devmajor,devminor) makedev(devmajor,devminor)
3 warnings generated.
./flist.c:473: error: undefined reference to 'makedev'
./flist.c:454: error: undefined reference to 'major'
./flist.c:457: error: undefined reference to 'major'
./flist.c:458: error: undefined reference to 'minor'
./flist.c:467: error: undefined reference to 'major'
./flist.c:467: error: undefined reference to 'makedev'
./flist.c:617: error: undefined reference to 'major'
./flist.c:619: error: undefined reference to 'minor'
./flist.c:621: error: undefined reference to 'minor'
./flist.c:788: error: undefined reference to 'makedev'
./flist.c:869: error: undefined reference to 'makedev'
./flist.c:1027: error: undefined reference to 'minor'
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [obj/local/armeabi-v7a/rsync] Error 1
… you probably need to update
/* #undef MAJOR_IN_SYSMACROS */
#define MAJOR_IN_SYSMACROS 1
Reading the project’s documentation doesn’t really help in building your own CA, but searching the Internet I found Fernando Barillas’ blog explaining how to create your own root certificate and how to create intermediate certificates from this.
I took it a step further I wrote a script generating new certificates for several services with different intermediates and possibly different configurations (e.g. depending on your distro and services certain cyphers (e.g. using ECC) may not be supported).
I also streamlined generating service specific key, cert and chain files. 😀
Have a look at the full Gist or just the most interesting part:
You’ll still have to deploy them yourself.
Fixed some issues with this Gist.
- Fixed a bug where intermediate CA certificates weren’t marked as CAs any more
- Updated the example CSRs and the script so it can now be run without errors
- Cleaned up
renew-certs.shby extracting functions for generating root CA, intermediate CA and service keys.
I tried to build a service monitor having the following features:
- showing the reachability of HTTP servers
- plotting the amount of messages in a specific RabbitMQ queue
- plotting the amount of queues with specific prefixes
- showing the status of RabbitMQ queues i.e. how many messages are in there? are there any consumers? are they hung?
- showing the availability of certain Redis clients
An example dashboard: