If you were to design a threading library today how would it look like? David Beazley manages to demonstrate a lot of edge cases in tiny examples … while live-coding! 🏆
If you use Python‘s Bottle micro-framework there’ll be a time where you’ll want to add custom plugins. To get a better feeling on what code gets executed when, I created a minimal Bottle app with a test plugin that logs what code gets executed. I uesed it to test both global and route-specific plugins.
When Python loads the module you’ll see that the plugins’
__init__()
and
setup()
methods will be called immediately when they are installed on the app or applied to the route. This happens in the order they appear in the code. Then the app is started.
The first time a route is called Bottle executes the plugins’
apply()
methods. This happens in “reversed order” of installation (which makes sense for a nested callback chain). This means first the route-specific plugins get applied then the global ones. Their result is cached, i.e. only the inner/wrapped function is executed from here on out.
Then for every request the
apply()
method’s inner function is executed. This happens in the “original” order again.
Below you can see the code and example logs for two requests. You can also clone the Gist and do your own experiments.
Wondering what the lifecycle of a plugin in #Python's #Bottle framework was I wrote a mini app to find out.http://t.co/rrhhYldRG5
— Riyad Preukschas (@riyadpr) July 5, 2015
During PyCon 2014 Erik Rose gave a very insightful talk about dos and don’ts of designing APIs. Towards the end he “gets meta” and groups all his points into categories drawing connections how different design goals influence each other. You see two main groups–”lingual” and “mathematical”–and he closes with this gem: 😀
This spotlights something that programming languages have over ordinary human languages. Programs are alive! They not only mean things when people read them, but they actually do things when run. So, very literally a program with carefully chosen symbols is poetry in motion.
— Erik Rose (PyCon 2014)
If you write software in Python you come to a point where you are testing a piece of code that expects a more or less elaborate dictionary as an argument to a function. As a good software developer we want that code properly tested but we want to use minimal fixtures to accomplish that.
So, I was looking for something that behaves like a dictionary, that you can give explicit return values for specific keys and that will give you some sort of a “default” return value when you try to access an “unknown” item (I don’t care what as long as there is no Exception raised (e.g. KeyError )).
My first thought was “why not use MagicMock?” … it’s a useful tool in so many situations.
1 2 | from mock import MagicMock m = MagicMock(foo="bar") |
But using MagicMock where dict is expected yields unexpected results.
1 2 3 4 5 6 | >>> # this works as expected >>> m.foo 'bar' >>> # but this doesn't do what you'd expect >>> m["foo"] <MagicMock name='mock.__getitem__()' id='4396280016'> |
First of all attribute and item access are treated differently. You setup MagicMock using key word arguments (i.e. “dict syntax”), but have to use attributes (i.e. “object syntax”) to access them.
Then I thought to yourself “why not mess with the magic methods?” __getitem__ and __getattr__ expect the same arguments anyway. So this should work:
1 2 | m = MagicMock(foo="bar") m.__getitem__.side_effect = m.__getattr__ |
Well? …
1 2 3 4 | >>> m.foo 'bar' >>> m["foo"] <MagicMock name='mock.foo' id='4554363920'> |
… No!
By this time I thought “I can’t be the first to need this” and started searching in the docs and sure enough they provide an example for this case.
1 2 3 4 | d = dict(foo="bar") m = MagicMock() m.__getitem__.side_effect = d.__getitem__ |
Does it work? …
1 2 3 4 5 6 7 8 9 10 | >>> m["foo"] 'bar' >>> m["bar"] Traceback (most recent call last): File "<stdin>", line 1, in <module> File ".../env/lib/python2.7/site-packages/mock.py", line 955, in __call__ return _mock_self._mock_call(*args, **kwargs) File ".../env/lib/python2.7/site-packages/mock.py", line 1018, in _mock_call ret_val = effect(*args, **kwargs) KeyError: 'bar' |
Well, yes and no. It works as long as you only access those items that you have defined to be in the dictionary. If you try to access any “unknown” item you get a KeyError .
After trying out different things the simplest answer to accomplish what I set out to do seems to be sub-classing defaultdict.
1 2 3 4 5 6 | from collections import defaultdict class MagicDict(defaultdict): def __missing__(self, key): result = self[key] = MagicDict() return result |
And? …
1 2 3 4 5 6 7 8 | >>> m["foo"] 'bar' >>> m["bar"] defaultdict(None, {}) >>> m.foo Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'MagicDict' object has no attribute 'foo' |
Indeed, it is. 😀
Well, not quite. There are still a few comfort features missing (e.g. a proper __repr__ ). The whole, improved and tested code can be found in this Gist:
https://alpha.app.net/riyad/post/12827299
https://alpha.app.net/riyad/post/12202500
I came up with a IMHO nice piece of code while working with and getting to know Python.
1 2 3 4 5 | incomplete_items = [ item.quantity_ordered > item.quantity_delivered for item in order.items ] if any(incomplete_items): do_something() |
This feels clean and obvious. It might not be very efficient though. :/
1 2 3 4 5 | has_incomplete_items = sum( item.quantity_ordered > item.quantity_delivered for item in order.items ) if has_incomplete_items: do_something() |
Doing it this way can be more efficient, since it can leverage generators and doesn’t need to go through the list again with any . But using sum over booleans feels hackish and non-obvious … 🙁
Thanks to @immoralist I’ve learned a new Python testing trick. I didn’t know about the “spec” argument for MagicMock. m(
Let’s see an example:
1 2 3 4 5 6 7 | from mock import MagicMock class SomeModel(object): foo = "f**" bar = "b**" m = MagicMock(spec=SomeModel) |
Here we create a mock object which mimics the interface of SomeModel as we would expect, returning mock values for things we access.
1 2 3 4 | >>> m.foo <MagicMock name='mock.foo' id='4506756880'> >>> m.bar <MagicMock name='mock.bar' id='4506754192'> |
Let’s see what happens if we call something else:
1 2 3 4 5 6 | >>> m.baz Traceback (most recent call last): File "<stdin>", line 1, in <module> File ".../env/lib/python2.7/site-packages/mock.py", line 658, in __getattr__ raise AttributeError("Mock object has no attribute %r" % name) AttributeError: Mock object has no attribute 'baz' |
It will fail loudly while a mock object without a spec would have returned a mock value as it did in the previous example.
But the magic doesn’t end there. You can still set additional attributes/methods “by hand” and have them not fail even if they aren’t part of the original spec.
1 2 3 | >>> m.baz = "bazzzzz" >>> m.baz 'bazzzzz' |
Learning new things makes me happy. 😀
In Python there is a simple way to make methods behave like properties using the @property decorator. But this only covers the getter side of things. What if you want to have a setter function for this “property”? Well there is a way. 🙂
Consider the following example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | import json class SomeModel(object): _foo = '{"foo":["bar", "baz"]}' @property def foo(self): return json.loads(self._foo) @foo.setter def foo_setter(self, new_value): self._foo = json.dumps(new_value) m = SomeModel() |
Now you can use the foo() method like a property.
1 2 | >>> m.foo {u'foo': [u'bar', u'baz']} |
This is a simple way to have a property contain a JSON string but access it as a Python dict, doing (de-)serialization on the fly.
So what if you want to set the value using a dict?
1 2 3 4 | >>> m.foo = ["foo", "bar"] Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: can't set attribute |
This is can easily trip up even seasoned Python developers. I’ve read code that did exactly this and I (as a novice) had to find out why the code failed. m(
The solution is quite simple … but “non-obvious” (as in: I wouldn’t have thought of that without consulting the docs) 🙁
1 2 3 | @foo.setter def foo(self, new_value) self._foo = json.dumps(new_value) |
Notice the method name? The setter and the getter methods have to have the same name!
Raymond Hettinger gives a great, concise and immediately useful examples of idiomatic Python.
