Introduction to metaclasses
This post describes my approach to explaining metaclasses. It is not about how to use metaclasses and its value is only educational. At the end of this post you should be able to define and reason about your metaclasses in python.
Metaclasses are an instance of metaprogramming which is available in some programming languages. To discuss metaclasses we first have to be familiar with classes. Consider this class definitions in python.
These class definitions make the names Spam, Egg and Ham available to the program. You have written them as the text of your program and the rest of your program can access them but consider this example.
This program is familiar. We have used the class names to create objects and placed them in today’s menu. Consider this other program.
The difference is that we have not used the class names to create objects. This time we have the value associated to a class name.
How does this program goes from its text representation to its runtime representation? Is it by value substitution? Is it by reference?
The short answer is that the names Spam, Egg, Ham are available in the current namespace and their
value is respectively
<class '__main__.Egg'>, and
The runtime representation of the program was obtained by value substitution just like your
everyday python. This is interesting because we have defined a name, value binding but the way
we have done this is quite different from the everyday syntax
name = value. What is going on?
Let’s take a brief detour to a very different world where you do not have classes and objects available in the language. Let’s go visit C.
Here is the initial example revisited. It’s not the same program and it does not work
but at least it look similar. Another thing you may notice is that we are prevented from putting
together in the same array different stuff. Which type should we put in place of
? in this program?
Let’s fix this. As all C types care about is the size of the thing we use an union to wrap them all.
If you knew beforehand all the possible choices for a menu item you could do this encoding in C
and achieve feature parity with our python example. But if you don’t know then you cannot. In C
you cannot change the members of an
union at runtime.
When you cannot achieve something using a feature of the language it is the time to make your own
implementation of it with structs and functions. What you would need is a
make_class function in
conjunction with a
This achieves feature parity for all we care. Now we can discuss the interesting part.
make_object are functions and are usually computed at runtime. This enables
you to write object oriented code in C. As we would like to show off to those pythonistas we
can even make some macros to emulate the syntax.
We now bid C farewell. It served us right and we have this idea that a
make_object functions are all we need. But how to do anything like this in Python?
What is the
make_class equivalent and is
__init__ method I have to write
all the times?
TODO(edoput) make_class for python
For all our purposes
make_class is replaced by
type. You may have used
type before to
inspect an object and find out what class it was created from but today we use it to create
classes. If we replace the class definitions using
type we can achieve feature parity.
We have now answered the first question and found that there is an alternative to define classes in python. This alternative uses ideas from everyday python such as variable assignment and function call to define classes but we still have much to explore. Here is a short list of unanswered questions.
- How do I define attributes?
- How do I define methods?
- How do I define a subclass?
These questions are immediate given our familiarity with the python class syntax. Since we have been taught the class syntax we have put attributes and methods in the class body and the name of the super class in parenthesis after the class name.
How do we translate this examples in the function and variable assignment style?
As you can see there are two extra arguments to
type that we ignored until now.
The second argument defines the classes from which to inherit and the third
argument maps the name bindings in the class body to a dictionary.
This is all we need to define classes at runtime. 1
TODO(edoput) the next step after type
Not everybody wants to only define classes at runtime using
type. People have
class syntax for a long time now and we can’t sweep it up from under their butt
withouth them noticing. There may be people that are downright opposed to
defining classes at runtime and we would like to accomodate them all.
To do so we have to make it possible to go from python’s class syntax to our function call convention. We will do this first with an external class syntax that python does not recognize and then we will acknowledge the internal class syntax.
We start with defining classes in JSON. This will limit what we can do
but it will be enough for our example. Because we want to do something more other
type we will also add an attribute representing the JSON source
to the new class.
This example’s interesting part is not the fact that we are
defining a class in JSON notation but the fact that now we have an alternative
type and the
class syntax. As long as we have a procedure from an input source
type we can define a class. We have abstracted this procedure with a function so that
we can reuse it as we please and this is the interesting bit.
Python’s internal syntax
In our previous example we have started with a description of the target class
in an external syntax 2, we have parsed it and fed the result to
type. Apart from
our choice of external syntax this is exactly what python does when it encounters
a class definition. It parses the syntax, finds name, bases and class body and feeds the
type after evaluation.
If we had a procedure to take in python syntax we could write our class
definitions in python, parse them and feed the result to
type. Just like
python does it!
While we could rig up something with the
ast module we are after a bigger fish.
What if we could specify a class using the class syntax and then ask python
to please use our procedure instead of
type? The example practically writes itself.
The intersting bits of this example are two:
- We define a procedure to create python classes. 3
- We ask python to use our
make_classprocedure, basically hijacking python’s everyday life and swapping it out for something that we wrote.
I’m not sure which bit is more mind-blowing. The first is very pleasing to the mind but the second really screams for attention as we wrangle python in doing our bidding even when we use python’s own syntax.
To make a very long story short it’s not common to use functions for this use case. The supported way to implement it is to go through classes and methods.
The zoo of metaclasses
The simplest example we can make reuses
type as a metaclass. While it does nothing different than
python’s builtin metaclass does, it shows how to use the
metaclass keyword in a class definition.
Once we have this we can finally give our metaclass a name.
Next we start writing the method used by the python interpreter when it will use our metaclass
When we feed this example to the python interpreter you will see an output similar to this.
Now is the moment to tie together everything we discussed. There is not much difference between
type and our own
make_class. They all
receive a name, a list of base classes, and a dictionary of class properties.
Moreover it does not look like
Meta has any interesting properties. The
is basically static and we use the name
Meta as a namespace. For all we care
Meta.__new__ is the function
make_class using the internal python syntax.
We have finally reached the end of this post. If you have understood everything you should be able to understand how to go from a class definition to a class object using the following.
typeand variable assignment.
- The class syntax native to python.
- Using a class implementing
You should also have understood that metaclasses in python are basically functions from
the internal syntax to an object by way of (eventually)
This means that you have at your disposal the full power of python while creating a class. This makes for a lot of power but it is unclear what to do with it now. My interpretation of this confusion follows.
The confusion comes from an intuitive division for your program’s lifecycle. In your mind the program goes through a static phase and a dynamic phase, e.g. is first read and then executed. Metaclasses instead pierce through this division and reveal that it was a continuum all this time.