Python: Zero to Hero
Home/Testing and Code Quality
Share

Chapter 37: Metaprogramming

Metaprogramming means writing code that operates on code. Instead of writing ten similar classes by hand, you write one function that generates all ten. Instead of repeating validation logic across twenty methods, you write a decorator that injects it everywhere at once. Instead of maintaining boilerplate, you eliminate it.

Python's metaprogramming tools are unusually accessible. You've already used some — decorators, dataclasses, @property — without realizing they're metaprogramming. This chapter goes further: class factories, dynamic attribute generation, exec/eval, and runtime code generation.

What Metaprogramming Is (and Isn't)

Metaprogramming is not magic or clever tricks. It's a tool for reducing repetition at the structural level — when copy-paste would multiply across classes or modules, not just loops.

Use it when:

  • You have N similar classes that differ only in a few parameters
  • You need to enforce a pattern across many functions or methods
  • You're building a framework or library, not a single application
  • Boilerplate is so repetitive that maintaining it manually is error-prone

Don't use it when:

  • A simple function or loop would do
  • The generated code would be harder to debug than the repetition
  • Your team won't understand it without significant explanation

type() as a Class Factory

In Chapter 36 you saw that type("Name", bases, namespace) creates a class dynamically. This is the core of class factories:

# Statically defined
class Dog:
    sound = "Woof"
    def speak(self):
        return f"{type(self).__name__} says {self.sound}"

# Dynamically equivalent
Dog = type("Dog", (object,), {
    "sound": "Woof",
    "speak": lambda self: f"{type(self).__name__} says {self.sound}",
})

print(Dog().speak())   # Dog says Woof

Building an animal factory

def make_animal(name: str, sound: str, legs: int):
    """Create an animal class dynamically."""
    def speak(self):
        return f"{self.name} says {self.sound}!"

    def __init__(self, name):
        self.name = name

    def __repr__(self):
        return f"{type(self).__name__}({self.name!r})"

    return type(name, (object,), {
        "sound":    sound,
        "legs":     legs,
        "__init__": __init__,
        "speak":    speak,
        "__repr__": __repr__,
    })


Dog  = make_animal("Dog",  "Woof", 4)
Cat  = make_animal("Cat",  "Meow", 4)
Bird = make_animal("Bird", "Tweet", 2)

fido = Dog("Fido")
whiskers = Cat("Whiskers")

print(fido.speak())       # Fido says Woof!
print(whiskers.speak())   # Whiskers says Meow!
print(Bird.legs)          # 2
print(isinstance(fido, Dog))   # True
print(isinstance(fido, Cat))   # False

A model factory for database tables

def make_model(table_name: str, fields: dict[str, type]):
    """
    Create a simple model class for a database table.
    fields: {"column_name": python_type}
    """
    def __init__(self, **kwargs):
        for name, expected_type in fields.items():
            value = kwargs.get(name)
            if value is not None and not isinstance(value, expected_type):
                raise TypeError(
                    f"{name} must be {expected_type.__name__}, "
                    f"got {type(value).__name__}"
                )
            setattr(self, name, value)

    def __repr__(self):
        attrs = ", ".join(f"{k}={getattr(self, k)!r}" for k in fields)
        return f"{type(self).__name__}({attrs})"

    def to_dict(self):
        return {k: getattr(self, k) for k in fields}

    @classmethod
    def from_dict(cls, data: dict):
        return cls(**{k: data.get(k) for k in fields})

    namespace = {
        "__init__":    __init__,
        "__repr__":    __repr__,
        "to_dict":     to_dict,
        "from_dict":   from_dict,
        "_table_name": table_name,
        "_fields":     fields,
    }
    return type(table_name.capitalize(), (object,), namespace)


# Create model classes without repeating yourself
User    = make_model("users",    {"name": str,  "email": str, "age": int})
Product = make_model("products", {"name": str,  "price": float, "stock": int})
Order   = make_model("orders",   {"user_id": int, "total": float})

alice = User(name="Alice", email="alice@example.com", age=30)
book  = Product(name="Python Book", price=29.99, stock=100)
order = Order(user_id=1, total=59.98)

print(alice)
print(book)
print(alice.to_dict())

Output:

Users(name='Alice', email='alice@example.com', age=30)
Products(name='Python Book', price=29.99, stock=100)
{'name': 'Alice', 'email': 'alice@example.com', 'age': 30}

Class Decorators

Just as function decorators wrap functions, class decorators wrap classes. A class decorator takes a class and returns a modified class:

def add_repr(cls):
    """Add a __repr__ based on __init__ parameter names."""
    import inspect
    params = list(inspect.signature(cls.__init__).parameters.keys())[1:]  # skip self

    def __repr__(self):
        attrs = ", ".join(f"{p}={getattr(self, p)!r}" for p in params
                          if hasattr(self, p))
        return f"{type(self).__name__}({attrs})"

    cls.__repr__ = __repr__
    return cls


def add_eq(cls):
    """Add __eq__ based on all instance attributes."""
    def __eq__(self, other):
        if type(self) is not type(other):
            return NotImplemented
        return self.__dict__ == other.__dict__

    cls.__eq__ = __eq__
    return cls


def frozen(cls):
    """Make instances immutable after __init__."""
    original_setattr = cls.__setattr__ if hasattr(cls, "__setattr__") else object.__setattr__
    original_init    = cls.__init__

    def __init__(self, *args, **kwargs):
        object.__setattr__(self, "_initialized", False)
        original_init(self, *args, **kwargs)
        object.__setattr__(self, "_initialized", True)

    def __setattr__(self, name, value):
        if object.__getattribute__(self, "_initialized"):
            raise AttributeError(f"{type(self).__name__} instances are immutable")
        object.__setattr__(self, name, value)

    cls.__init__    = __init__
    cls.__setattr__ = __setattr__
    return cls


@add_repr
@add_eq
@frozen
class Point:
    def __init__(self, x: float, y: float):
        self.x = x
        self.y = y


p1 = Point(1.0, 2.0)
p2 = Point(1.0, 2.0)
p3 = Point(3.0, 4.0)

print(p1)           # Point(x=1.0, y=2.0)
print(p1 == p2)     # True
print(p1 == p3)     # False

try:
    p1.x = 99       # AttributeError: Point instances are immutable
except AttributeError as e:
    print(e)

__init_subclass__ — Framework Hooks

__init_subclass__ is called whenever a subclass is defined. It's one of the cleanest ways to build registration systems and enforce constraints on subclasses:

class Serializable:
    """
    Mix-in that requires subclasses to declare _fields
    and auto-generates to_dict / from_dict.
    """
    _fields: tuple[str, ...] = ()

    def __init_subclass__(cls, fields: tuple[str, ...] = (), **kwargs):
        super().__init_subclass__(**kwargs)
        if fields:
            cls._fields = fields
        elif not cls._fields:
            raise TypeError(
                f"{cls.__name__} must declare fields=(...) or set _fields"
            )

        def to_dict(self) -> dict:
            return {f: getattr(self, f) for f in cls._fields}

        @classmethod
        def from_dict(klass, data: dict):
            return klass(**{f: data[f] for f in klass._fields})

        cls.to_dict   = to_dict
        cls.from_dict = from_dict


class User(Serializable, fields=("name", "email", "age")):
    def __init__(self, name, email, age):
        self.name  = name
        self.email = email
        self.age   = age


class Product(Serializable, fields=("name", "price", "stock")):
    def __init__(self, name, price, stock):
        self.name  = name
        self.price = price
        self.stock = stock


alice = User("Alice", "alice@example.com", 30)
print(alice.to_dict())
# {'name': 'Alice', 'email': 'alice@example.com', 'age': 30}

alice2 = User.from_dict({"name": "Alice", "email": "alice@example.com", "age": 30})
print(alice2.name)   # Alice

book = Product("Python Book", 29.99, 100)
print(book.to_dict())
# {'name': 'Python Book', 'price': 29.99, 'stock': 100}

Dynamic Attribute Generation

Sometimes attributes don't exist statically — they're computed or looked up at access time:

class DynamicConfig:
    """
    Config object where any attribute is read from a dict.
    Nested dicts become nested DynamicConfig objects.
    """
    def __init__(self, data: dict):
        object.__setattr__(self, "_data", data)

    def __getattr__(self, name: str):
        data = object.__getattribute__(self, "_data")
        if name not in data:
            raise AttributeError(f"No config key {name!r}")
        value = data[name]
        if isinstance(value, dict):
            return DynamicConfig(value)
        return value

    def __setattr__(self, name: str, value):
        data = object.__getattribute__(self, "_data")
        data[name] = value

    def __repr__(self):
        data = object.__getattribute__(self, "_data")
        return f"DynamicConfig({data!r})"


config = DynamicConfig({
    "database": {
        "host":     "localhost",
        "port":     5432,
        "name":     "myapp",
    },
    "cache": {
        "backend":  "redis",
        "timeout":  300,
    },
    "debug": True,
})

print(config.debug)                  # True
print(config.database.host)          # localhost
print(config.database.port)          # 5432
print(config.cache.backend)          # redis

config.debug = False
print(config.debug)                  # False

Auto-generating methods

class AutoProperty:
    """
    Metaclass-free way to generate properties from a list of field names.
    """

    def __init_subclass__(cls, fields=(), **kwargs):
        super().__init_subclass__(**kwargs)

        for field in fields:
            private = f"_{field}"

            def make_property(priv):
                @property
                def prop(self):
                    return getattr(self, priv, None)

                @prop.setter
                def prop(self, value):
                    setattr(self, priv, value)

                return prop

            setattr(cls, field, make_property(private))


class Point(AutoProperty, fields=("x", "y", "z")):
    def __init__(self, x, y, z=0):
        self._x = x
        self._y = y
        self._z = z

    def __repr__(self):
        return f"Point({self.x}, {self.y}, {self.z})"


p = Point(1, 2, 3)
print(p)       # Point(1, 2, 3)
p.x = 10
print(p)       # Point(10, 2, 3)

exec and eval — Runtime Code Execution

eval evaluates a single expression. exec executes statements. Both should be used carefully — arbitrary code execution is a security risk if the input comes from untrusted sources.

# eval — evaluate an expression, returns a value
result = eval("2 ** 10")
print(result)   # 1024

# With a custom namespace
ns = {"x": 5, "y": 3}
print(eval("x ** y", ns))   # 125

# exec — execute statements (returns None)
code = """
def greet(name):
    return f"Hello, {name}!"
"""
namespace = {}
exec(code, namespace)
print(namespace["greet"]("Alice"))   # Hello, Alice!

Safe use: controlled namespace

# Only expose specific names — block access to builtins
safe_ns = {
    "__builtins__": {},   # no built-ins
    "abs":   abs,
    "round": round,
    "min":   min,
    "max":   max,
}

formula = "max(abs(-5), round(3.7))"
result  = eval(formula, safe_ns)
print(result)   # 5

# Dangerous — never do this with untrusted input
# eval("__import__('os').system('rm -rf /')")

Generating code at runtime

def make_validator_class(rules: dict[str, tuple]):
    """
    Generate a validator class from a rules dict.
    rules: {"field": (type, min_val, max_val)}
    """
    method_lines = [
        "class GeneratedValidator:",
        "    def validate(self, data):",
        "        errors = []",
    ]

    for field, (expected_type, min_val, max_val) in rules.items():
        method_lines += [
            f"        if '{field}' not in data:",
            f"            errors.append('{field} is required')",
            f"        elif not isinstance(data['{field}'], {expected_type.__name__}):",
            f"            errors.append('{field} must be {expected_type.__name__}')",
        ]
        if min_val is not None:
            method_lines += [
                f"        elif data['{field}'] < {min_val}:",
                f"            errors.append('{field} must be >= {min_val}')",
            ]
        if max_val is not None:
            method_lines += [
                f"        elif data['{field}'] > {max_val}:",
                f"            errors.append('{field} must be <= {max_val}')",
            ]

    method_lines += [
        "        return errors",
    ]

    source = "\n".join(method_lines)
    ns: dict = {}
    exec(source, ns)
    return ns["GeneratedValidator"]


UserValidator = make_validator_class({
    "age":    (int,   0,   150),
    "score":  (float, 0.0, 100.0),
    "name":   (str,   None, None),
})

v = UserValidator()
print(v.validate({"name": "Alice", "age": 30, "score": 95.5}))   # []
print(v.validate({"age": -1, "score": 200.0}))
# ["name is required", "age must be >= 0", "score must be <= 100.0"]

__class_getitem__ — Making Classes Generic

__class_getitem__ is called when you write MyClass[SomeType]. This is how list[int], dict[str, int], and custom generics work:

from typing import Generic, TypeVar

T = TypeVar("T")

class TypedList(Generic[T]):
    """A list that enforces a single element type."""

    def __init__(self, item_type: type):
        self._type  = item_type
        self._items: list = []

    def __class_getitem__(cls, item_type):
        """Called when you write TypedList[int]."""
        instance = cls.__new__(cls)
        instance.__init__(item_type)
        return instance

    def append(self, item):
        if not isinstance(item, self._type):
            raise TypeError(
                f"Expected {self._type.__name__}, got {type(item).__name__}"
            )
        self._items.append(item)

    def __iter__(self):
        return iter(self._items)

    def __len__(self):
        return len(self._items)

    def __repr__(self):
        return f"TypedList[{self._type.__name__}]({self._items!r})"


int_list  = TypedList[int]
str_list  = TypedList[str]

int_list.append(1)
int_list.append(2)
int_list.append(3)
print(int_list)   # TypedList[int]([1, 2, 3])

try:
    int_list.append("hello")   # TypeError
except TypeError as e:
    print(e)

__set_name__ — Descriptor Self-Registration

__set_name__ is called on a descriptor when the class that contains it is defined. It tells the descriptor its own name:

class Validated:
    """A descriptor that validates on set using a provided validator function."""
    def __init__(self, validator, doc=""):
        self.validator = validator
        self.__doc__   = doc
        self.name      = None     # set by __set_name__

    def __set_name__(self, owner, name):
        self.name = name
        # Register this descriptor with the owner class
        if not hasattr(owner, "_validated_fields"):
            owner._validated_fields = []
        owner._validated_fields.append(name)

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return obj.__dict__.get(self.name)

    def __set__(self, obj, value):
        error = self.validator(value)
        if error:
            raise ValueError(f"{self.name}: {error}")
        obj.__dict__[self.name] = value


# Validator functions — return error string or None
def positive(v):
    return "must be positive" if v <= 0 else None

def non_empty_string(v):
    if not isinstance(v, str):
        return "must be a string"
    if not v.strip():
        return "must not be empty"
    return None

def valid_email(v):
    import re
    if not re.match(r"[^@]+@[^@]+\.[^@]+", v):
        return "must be a valid email"
    return None


class User:
    name  = Validated(non_empty_string, "User's display name")
    email = Validated(valid_email,      "User's email address")
    age   = Validated(positive,         "User's age in years")

    def __init__(self, name, email, age):
        self.name  = name
        self.email = email
        self.age   = age

    def __repr__(self):
        return f"User({self.name!r}, {self.email!r}, age={self.age})"


alice = User("Alice", "alice@example.com", 30)
print(alice)
print(f"Validated fields: {User._validated_fields}")

try:
    bad = User("", "not-an-email", -5)
except ValueError as e:
    print(f"Error: {e}")

Output:

User('Alice', 'alice@example.com', age=30)
Validated fields: ['name', 'email', 'age']
Error: name: must not be empty

Building a Mini ORM with Metaprogramming

Let's put it all together — a small ORM (Object-Relational Mapper) that uses metaprogramming to map Python classes to database tables:

"""
mini_orm.py — A small ORM built with metaprogramming techniques.
"""
import sqlite3
from typing import Any, Optional


# ── Field descriptors ──────────────────────────────────────────────────────────

class Field:
    """Base field descriptor for ORM models."""
    sql_type = "TEXT"

    def __init__(self, required: bool = True, default: Any = None, unique: bool = False):
        self.required = required
        self.default  = default
        self.unique   = unique
        self.name     = None   # set by __set_name__

    def __set_name__(self, owner, name):
        self.name = name

    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return obj.__dict__.get(self.name, self.default)

    def __set__(self, obj, value):
        if value is None and self.required:
            raise ValueError(f"{self.name} is required")
        obj.__dict__[self.name] = value

    def validate(self, value):
        pass

    def column_def(self) -> str:
        parts = [self.name, self.sql_type]
        if self.unique:
            parts.append("UNIQUE")
        return " ".join(parts)


class IntField(Field):
    sql_type = "INTEGER"

    def __set__(self, obj, value):
        if value is not None and not isinstance(value, int):
            raise TypeError(f"{self.name} must be int, got {type(value).__name__}")
        super().__set__(obj, value)


class FloatField(Field):
    sql_type = "REAL"

    def __set__(self, obj, value):
        if value is not None and not isinstance(value, (int, float)):
            raise TypeError(f"{self.name} must be numeric")
        super().__set__(obj, float(value) if value is not None else None)


class TextField(Field):
    sql_type = "TEXT"

    def __init__(self, max_length: int = 0, **kwargs):
        super().__init__(**kwargs)
        self.max_length = max_length

    def __set__(self, obj, value):
        if value is not None:
            if not isinstance(value, str):
                raise TypeError(f"{self.name} must be str")
            if self.max_length and len(value) > self.max_length:
                raise ValueError(f"{self.name} too long (max {self.max_length})")
        super().__set__(obj, value)


# ── Model metaclass ────────────────────────────────────────────────────────────

class ModelMeta(type):
    """Metaclass that collects Field descriptors and generates SQL helpers."""

    def __new__(mcs, name, bases, namespace):
        fields = {}

        # Inherit fields from base models
        for base in bases:
            if hasattr(base, "_fields"):
                fields.update(base._fields)

        # Collect Field instances from this class's namespace
        for attr_name, value in list(namespace.items()):
            if isinstance(value, Field):
                fields[attr_name] = value

        namespace["_fields"] = fields
        namespace["_table"]  = name.lower() + "s"   # simple pluralization

        cls = super().__new__(mcs, name, bases, namespace)
        return cls


# ── Base model ─────────────────────────────────────────────────────────────────

class Model(metaclass=ModelMeta):
    """Base class for all ORM models."""

    id = IntField(required=False)

    def __init__(self, **kwargs):
        for name, field in self._fields.items():
            value = kwargs.get(name, field.default)
            setattr(self, name, value)

    def __repr__(self):
        attrs = " ".join(
            f"{k}={getattr(self, k)!r}"
            for k in self._fields
            if k != "id" or getattr(self, k) is not None
        )
        return f"{type(self).__name__}({attrs})"

    # ── Schema ──────────────────────────────────────────────────────────────────

    @classmethod
    def create_table(cls, conn: sqlite3.Connection) -> None:
        columns = ["id INTEGER PRIMARY KEY AUTOINCREMENT"]
        for name, field in cls._fields.items():
            if name == "id":
                continue
            columns.append(field.column_def())
        sql = f"CREATE TABLE IF NOT EXISTS {cls._table} ({', '.join(columns)})"
        conn.execute(sql)
        conn.commit()

    # ── CRUD ────────────────────────────────────────────────────────────────────

    def save(self, conn: sqlite3.Connection) -> "Model":
        data = {k: getattr(self, k) for k in self._fields if k != "id"}
        if self.id is None:
            # INSERT
            cols   = ", ".join(data.keys())
            placeholders = ", ".join("?" for _ in data)
            cursor = conn.execute(
                f"INSERT INTO {self._table} ({cols}) VALUES ({placeholders})",
                list(data.values()),
            )
            self.id = cursor.lastrowid
        else:
            # UPDATE
            set_clause = ", ".join(f"{k} = ?" for k in data)
            conn.execute(
                f"UPDATE {self._table} SET {set_clause} WHERE id = ?",
                [*data.values(), self.id],
            )
        conn.commit()
        return self

    @classmethod
    def get(cls, conn: sqlite3.Connection, id: int) -> Optional["Model"]:
        row = conn.execute(
            f"SELECT * FROM {cls._table} WHERE id = ?", (id,)
        ).fetchone()
        if row is None:
            return None
        return cls._from_row(row)

    @classmethod
    def all(cls, conn: sqlite3.Connection) -> list["Model"]:
        rows = conn.execute(f"SELECT * FROM {cls._table}").fetchall()
        return [cls._from_row(r) for r in rows]

    @classmethod
    def filter(cls, conn: sqlite3.Connection, **conditions) -> list["Model"]:
        where  = " AND ".join(f"{k} = ?" for k in conditions)
        values = list(conditions.values())
        rows   = conn.execute(
            f"SELECT * FROM {cls._table} WHERE {where}", values
        ).fetchall()
        return [cls._from_row(r) for r in rows]

    def delete(self, conn: sqlite3.Connection) -> None:
        if self.id is None:
            raise ValueError("Cannot delete unsaved model")
        conn.execute(f"DELETE FROM {self._table} WHERE id = ?", (self.id,))
        conn.commit()
        self.id = None

    @classmethod
    def _from_row(cls, row: tuple) -> "Model":
        """Create an instance from a DB row tuple."""
        col_names = ["id"] + [k for k in cls._fields if k != "id"]
        data = dict(zip(col_names, row))
        return cls(**data)


# ── Define models ──────────────────────────────────────────────────────────────

class User(Model):
    name  = TextField(max_length=100)
    email = TextField(unique=True)
    age   = IntField(required=False, default=0)


class Post(Model):
    title   = TextField(max_length=200)
    content = TextField(required=False, default="")
    user_id = IntField()
    views   = IntField(required=False, default=0)


# ── Demo ──────────────────────────────────────────────────────────────────────

if __name__ == "__main__":
    conn = sqlite3.connect(":memory:")

    User.create_table(conn)
    Post.create_table(conn)

    # Create users
    alice = User(name="Alice", email="alice@example.com", age=30).save(conn)
    bob   = User(name="Bob",   email="bob@example.com",   age=25).save(conn)

    print(f"Created: {alice}")
    print(f"Created: {bob}")

    # Create posts
    p1 = Post(title="Hello World", content="My first post!", user_id=alice.id).save(conn)
    p2 = Post(title="Python Tips", content="Use list comprehensions.", user_id=alice.id, views=42).save(conn)
    p3 = Post(title="Databases",   content="SQL is powerful.", user_id=bob.id).save(conn)

    # Query
    all_users = User.all(conn)
    print(f"\nAll users: {all_users}")

    alice_posts = Post.filter(conn, user_id=alice.id)
    print(f"\nAlice's posts: {alice_posts}")

    # Update
    alice.age = 31
    alice.save(conn)
    refetched = User.get(conn, alice.id)
    print(f"\nUpdated Alice: {refetched}")

    # Delete
    p3.delete(conn)
    print(f"\nPosts after delete: {Post.all(conn)}")

    conn.close()

Output:

Created: User(name='Alice' email='alice@example.com' age=30)
Created: User(name='Bob' email='bob@example.com' age=25)

All users: [User(id=1 name='Alice' ...), User(id=2 name='Bob' ...)]

Alice's posts: [Post(id=1 title='Hello World' ...), Post(id=2 title='Python Tips' ...)]

Updated Alice: User(id=1 name='Alice' email='alice@example.com' age=31)

Posts after delete: [Post(id=1 ...), Post(id=2 ...)]

This ORM used:

  • __set_name__ on descriptors to auto-capture field names
  • ModelMeta metaclass to collect all Field instances into _fields
  • type() introspection to generate CREATE TABLE SQL automatically
  • Dynamic SQL generation for INSERT, UPDATE, SELECT, DELETE
  • Inheritance — Model provides all CRUD, subclasses just declare fields

What You Learned in This Chapter

  • Metaprogramming reduces structural repetition — use it when N similar classes or N similar method patterns would otherwise require copy-paste.
  • type(name, bases, namespace) creates classes dynamically at runtime.
  • Class decorators modify a class after definition — the same pattern as function decorators but applied to the class object.
  • __init_subclass__ runs when a subclass is defined — use it for registration, validation, and automatic method generation without metaclasses.
  • Dynamic attribute access with __getattr__ / __getattribute__ enables lazy loading, proxies, and configuration objects.
  • exec executes arbitrary Python statements; eval evaluates expressions. Both accept a custom namespace for safety. Never run untrusted input.
  • __class_getitem__ implements MyClass[Type] syntax for custom generic classes.
  • __set_name__ tells descriptors their own attribute name and enables self-registration.
  • Combining these tools — descriptors + metaclass + __init_subclass__ + dynamic SQL — you can build a full ORM in ~200 lines.

What's Next?

Chapter 38 is the final chapter — The Capstone Project. You'll build a complete, production-quality Python application from scratch: a REST API with authentication, a database, a background task queue, structured logging, tests, and a Dockerfile. Everything you've learned in this book comes together in one project.

© 2026 Abhilash Sahoo. Python: Zero to Hero.