Choose your next depth / OPTIONAL DEPTH

Iterators, generators & streaming

Process a sequence without requiring every record to be resident in memory.

Apply this to your Python data tools, PixelFed quality work, n8n integrations or BI delivery when the project needs it.

Understand the idea

A generator defers execution until the consumer asks for a value. That can reduce the input retained in memory, but an unbounded grouping dictionary can still grow with the number of distinct keys. Streaming describes the whole design, not merely the presence of yield.

Keep file ownership clear. Opening a file outside a generator and returning an iterator after closing that file creates a reader that cannot be used. Test early termination as well as complete consumption.

What to learn

• Explain iterable versus iterator and one-pass consumption.
• Yield parsed rows while the file context is open.
• Use bounded batches and measure peak memory.

A small original example

import csv

def rows(path):
    with open(path, encoding="utf-8", newline="") as handle:
        for row in csv.DictReader(handle):
            yield row

Practice on your own data

Replace a fixture reader that returns list(reader) with a generator and compare peak memory on a larger synthetic file.

Practice this lesson in the Python lab →

Common pitfalls

• An iterator may be exhausted after the first pass.
• list(generator) materializes the records again.

Prerequisites

Loops & accumulation · Files, CSV, JSON & paths

Sources and further reading

• Python · iterators and generators ↗Official project documentation

Track progress on this topic → · Markdown source ↗