Skip unreadable files in awk scripts

Have you ever needed to give awk multiple files to process, but wanted it to skip files that it can’t open? Use this snippet!

BEGINFILE {
    # skip files that cannot be opened
    if ( ERRNO != "" ) {
        nextfile
    }
}

This trick is mentioned in the documentation for the BEGINFILE pattern, but not spelled out as code. You’re welcome. 😀

Scaffolding for fetching and parsing emails from IMAP with Python

A friend asked me if I could help him write a Python script for fetching and processing data from emails in his mailbox … Well, the thing with emails is that they’re a pain to work with (in any form). So, I tried to help him out with a little scaffolding (also available as a Gist).

#!/usr/bin/python3
#
# Author: Riyad Preukschas <riyad@informatik.uni-bremen.de>
# License: Mozilla Public License 2.0
#
# Scaffolding for fetching and parsing emails from IMAP
import imaplib
import email.parser
import email.policy
IMAP_HOST = ''
IMAP_USER = ''
IMAP_PASSWORD = ''
# connect to server
with imaplib.IMAP4_SSL(IMAP_HOST) as imap:
# login with user name and password
imap.login(IMAP_USER, IMAP_PASSWORD)
# select mail "folder" to work in
imap.select('INBOX', readonly=True)
# list ALL mails in folder ... can also be used to filter mails
# see https://tools.ietf.org/html/rfc2060#section-6.4.4
typ, data = imap.search(None, 'ALL')
message_ids = data[0].split()
for message_id in message_ids:
# fetch "raw" email data
typ, data = imap.fetch(message_id, '(RFC822)')
response_part_rfc822_data = data[0][1]
# create an EmailMessage object from the raw data
msg = email.parser.BytesParser(
policy=email.policy.default).parsebytes(response_part_rfc822_data)
# do your stuff here ...
# for access to specific parts of the email message
# see https://docs.python.org/3/library/email.message.html#email.message.EmailMessage
email_subject = msg['Subject']
email_from = msg['From']
# get email body in plain text form (instead of html)
# ... this might fail, because some mails are HTML-only :/
email_text = msg.get_body('plain').get_content()
print('From : ' + email_from + '\n')
print('Subject : ' + email_subject + '\n')
print(email_text)

Multi-step Refactoring Pains in C++

Titus Winters talks about maintaining and refactoring large C++ code bases (i.e. code bodies that require multi-step refactoring). He describes how “higher-level” language features effectively make refactoring harder (e.g. functions, classes, templates, concepts).

Abspielen
Das Video wird von Youtube eingebettet abespielt. Es gilt die Datenschutzerklärung von Google

Unsafe Chrome Sometimes Necessary

In my work – every now and then – I found myself in need of a browser with reduced security checks (mainly to gloss over cross domain XMLHttpRequests and SSL certificate violations) for testing purposes. I didn’t want to take the risk and use my main browser session with these settings, so I made me a script (also available as a Gist). 🙂

#
# Author: Riyad Preukschas <riyad@informatik.uni-bremen.de>
# License: Mozilla Public License 2.0
#
# Loads an unsafe (i.e. with several security features disabled) instance of
# Chrome with a temporary profile (i.e. all data is lost once Chrome is closed)
chrome-unsafe() {
# for Homebrew Cask (see http://caskroom.io/) compatibility
local -a CHROME_PATHS
CHROME_PATHS=( \
"/opt/homebrew-cask/Caskroom/google-chrome-dev/latest/Google Chrome.app/Contents/MacOS/Google Chrome" \
"/opt/homebrew-cask/Caskroom/google-chrome/latest/Google Chrome.app/Contents/MacOS/Google Chrome" \
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
"chrome" \
"chromium" \
)
local CHROME="${CHROME_PATHS[1]}"
while [[ -n CHROME_PATHS ]] && ! type ${CHROME} > /dev/null; do
shift CHROME_PATHS
CHROME="${CHROME_PATHS[1]}"
done
local DATA_DIR="$(mktemp -d -t 'chrome-unsafe_data_dir.XXXXXXXXXX')"
# for a listing of command line switches see:
# https://peter.sh/experiments/chromium-command-line-switches/
"${CHROME}" \
--disable-bundled-ppapi-flash \
--disable-popup-blocking \
--disable-sync \
--disable-web-security \
--ignore-certificate-errors \
--no-default-browser-check \
--no-first-run \
--non-secure \
--user-data-dir="${DATA_DIR}" \
$@ >/dev/null 2>&1 &!
# for now the window will not be brought to the front
}

Tip:
If you use oh my ZSH you can save this file in

~/.oh-my-zsh/custom/plugins/chrome-unsafe/chrome-unsafe.plugin.zsh

and add “chrome-unsafe” to your list of used plugins in

~/.zshrc

 

Summing Booleans For Fun And Profit

I came up with a IMHO nice piece of code while working with and getting to know Python.

incomplete_items = [
    item.quantity_ordered > item.quantity_delivered for item in order.items
]
if any(incomplete_items):
    do_something()

This feels clean and obvious. It might not be very efficient though. :/

has_incomplete_items = sum(
    item.quantity_ordered > item.quantity_delivered for item in order.items
)
if has_incomplete_items:
    do_something()

Doing it this way can be more efficient, since it can leverage generators and doesn’t need to go through the list again with

any

. But using 

sum

  over booleans feels hackish and non-obvious … 🙁

MagicMock With Spec

Thanks to @immoralist I’ve learned a new Python testing trick. I didn’t know about the “spec” argument for MagicMock. m(
Let’s see an example:

from mock import MagicMock

class SomeModel(object):
    foo = "f**"
    bar = "b**"

m = MagicMock(spec=SomeModel)

Here we create a mock object which mimics the interface of 

SomeModel

  as we would expect, returning mock values for things we access.

>>> m.foo
<MagicMock name='mock.foo' id='4506756880'>
>>> m.bar
<MagicMock name='mock.bar' id='4506754192'>

Let’s see what happens if we call something else:

>>> m.baz
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../env/lib/python2.7/site-packages/mock.py", line 658, in __getattr__
    raise AttributeError("Mock object has no attribute %r" % name)
AttributeError: Mock object has no attribute 'baz'

It will fail loudly while a mock object without a spec would have returned a mock value as it did in the previous example.

But the magic doesn’t end there. You can still set additional attributes/methods “by hand” and have them not fail even if they aren’t part of the original spec.

>>> m.baz = "bazzzzz"
>>> m.baz
'bazzzzz'

Learning new things makes me happy. 😀

Tripping Over Property Setters in Python

In Python there is a simple way to make methods behave like properties using the @property decorator. But this only covers the getter side of things. What if you want to have a setter function for this “property”? Well there is a way. 🙂
Consider the following example:

import json

class SomeModel(object):
  _foo = '{"foo":["bar", "baz"]}'

  @property
  def foo(self):
    return json.loads(self._foo)

  @foo.setter
  def foo_setter(self, new_value):
    self._foo = json.dumps(new_value)

m = SomeModel()

Now you can use the 

foo()

  method like a property.

>>> m.foo
{u'foo': [u'bar', u'baz']}

This is a simple way to have a property contain a JSON string but access it as a Python dict, doing (de-)serialization on the fly.
So what if you want to set the value using a dict?

>>> m.foo = ["foo", "bar"]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: can't set attribute

This is can easily trip up even seasoned Python developers. I’ve read code that did exactly this and I (as a novice) had to find out why the code failed. m(
The solution is quite simple … but “non-obvious” (as in: I wouldn’t have thought of that without consulting the docs) 🙁

  @foo.setter
  def foo(self, new_value)
    self._foo = json.dumps(new_value)

Notice the method name? The setter and the getter methods have to have the same name!

Old Games’ Source Code

Fabien Sanglard has written an interesting drill down into the recently released source code for Doom3. He tries to reason why certain things are the way they are and also had a few of his questions answered by John Carmack himself.

There was news about the source code for the Prince of Persia game on Apple II computers restored from a set of ancient floppy disks. There is also a great quote on why going through the trouble of restoring/releasing the source code of those games is something worthwhile.

“Because if we didn’t, it might have disappeared forever.”

Video game source code is a bit like the sheet music to a piano sonata that’s already been performed and recorded. One might reasonably ask: If you have the recording, what do you need the sheet music for?

You don’t, if all you want is to listen and enjoy the music. But to a pianist performing the piece, or a composer who wants to study it or arrange it for different instruments, the original score is valuable.

It’s possible, up to a point, to reverse-engineer new source code from a published video game, much as a capable musician can transcribe a musical score from listening to a performance. But in both cases, there’s no substitute for the original document as a direct line to the creator’s intentions and work process. As such, it has both practical and historical value, to the small subset of the game-playing/music-listening community that cares.