098. Generator Expressions

The irony is not lost on Rohan: his RAM manager is itself using too much RAM.

He stores 7 days of snapshots — about 100,000 entries. To find the maximum RAM usage:

# Version 1 — list comprehension: builds 100,000-element list first
max_usage = max([snap["percent"] for snap in history])

# Version 2 — generator expression: produces values one at a time, O(1) memory
max_usage = max(snap["percent"] for snap in history)

The generator expression passes values to max() one at a time. At no point does Python hold 100,000 floats in memory simultaneously.

He also computes total MB used across all processes in the latest snapshot:

total_mb = sum(proc["rss_mb"] for proc in snapshot["processes"])

And finds the first process exceeding 500 MB without scanning the whole list:

big = next(
    (proc for proc in snapshot["processes"] if proc["rss_mb"] > 500),
    None    # default if none found
)

next() stops the generator at the first match. If Chrome is the third process and there are 200 processes, 197 are never even examined.

Generator vs list comprehension: Use (...) (generator) when you pass it directly into sum(), max(), any(), all(), or next(). Use [...] (list) when you need to iterate multiple times, index into it, or check its length.

💡 Fun fact: Generator expressions were added in Python 2.4 (2004) in PEP 289. Guido van Rossum said the main motivation was that sum(x*x for x in range(1000)) is so much more natural than sum([x*x for x in range(1000)]). Under the hood, a generator expression creates a generator object — the same thing you get from a function using yield.

⚠️ Watch out: Generators are single-use. Once exhausted, iterating again yields nothing. If you need to iterate a sequence more than once (e.g., compute both sum and max), either use a list comprehension or call list() on the generator first.

🤔 Think about it: If any(x > 0 for x in items) short-circuits at the first True, what happens with all(x > 0 for x in items) on an empty list? Does it return True or False, and why?

Learning objectives

Write generator expressions as memory-efficient alternatives to list comprehensions
Use next() with a default to get the first matching element
Combine generators with sum(), any(), all(), max()
Understand that generators are single-use and why that matters

Key concepts

generator expression
lazy evaluation
next()
memory efficiency

Try it

Concept detail

Generator expression syntax: (expression for variable in iterable [if condition])

Key difference from list comprehension: [xx for x in range(1000)] — builds a 1000-element list in memory immediately (xx for x in range(1000)) — creates a generator that yields values one at a time

Memory footprint: List comprehension: O(n) — all n values stored simultaneously Generator expression: O(1) — only one value in memory at a time

Works as argument to consuming functions: sum(x*x for x in range(n)) — no list created max(s[“percent”] for s in snaps) — streams values into max() any(x > 0 for x in items) — stops at first True (short-circuit) all(x > 0 for x in items) — stops at first False

next() with a default — get first match or fallback: next((x for x in items if x > threshold), None) — returns first match, or None if no match (default prevents StopIteration)

Generators are single-use: g = (x for x in [1, 2, 3]) list(g) → [1, 2, 3] list(g) → [] — already exhausted Use a list comprehension when you need to iterate more than once.

Count pattern (avoid len on generator): sum(1 for x in items if condition) — counts matches without building a list

Solution

def sum_of_squares(n):
    return sum(i * i for i in range(1, n + 1))

def first_over(numbers, threshold):
    return next((x for x in numbers if x > threshold), None)

def count_long_words(words, min_len):
    return sum(1 for w in words if len(w) > min_len)

Tests

def test_sum_of_squares():
    assert sum_of_squares(3) == 14  # 1 + 4 + 9
    assert sum_of_squares(5) == 55  # 1+4+9+16+25

def test_sum_of_squares_one():
    assert sum_of_squares(1) == 1

def test_first_over_found():
    assert first_over([1, 5, 3, 8, 2], 4) == 5

def test_first_over_not_found():
    assert first_over([1, 2, 3], 10) is None

def test_count_long_words():
    words = ["cat", "elephant", "dog", "rhinoceros", "ox"]
    assert count_long_words(words, 4) == 2

def test_count_long_words_none():
    assert count_long_words(["a", "bb"], 5) == 0