Why your Python script is slow: two lines took it 3.71 s to 1.20 s
One script, one million rows, one question: where does the time go? This is the write-up of the video above. Every number comes from the same measurement files the video used, all on one Mac mini (M4 Pro, 48 GB) with Python 3.14.6. The workload is synthetic: a deterministic generator (seed 42) writes a sales CSV so anyone can reproduce the run.
The result first
- report.py (baseline): 3.71 s — single run, timed inside the script
- report_cached.py (two lines added): 1.20 s — single run, timed inside the script
cmp report.txt report2.txt→ identical, byte for byte
About 3.1× faster, same output, two lines changed. The interesting part is how we found the two lines, and where the trick stops working.
What the script does
report.py reads a CSV of 1,000,000 sales rows, parses each row's date string, totals revenue by month and region, and writes a text report. We measure wall time, and the report must stay identical.
$ python3 report.py sales.csv report.txt wrote report.txt: 1,000,000 rows in 3.71 s
Not slow enough to notice once. Slow enough to hate inside a loop.
Ask Python where the time goes
$ python3 -m cProfile -s tottime report.py sales.csv report.txt | head -12
32016593 function calls (32016422 primitive calls) in 8.440 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1000000 3.045 0.000 5.556 0.000 _strptime.py:516(_strptime)
1000001 0.859 0.000 1.411 0.000 csv.py:173(__next__)
1000000 0.465 0.000 6.021 0.000 _strptime.py:812(_strptime_datetime_datetime)
1 0.451 0.451 6.947 6.947 report.py:12(summarize)
The top line is the date parser, strptime: one million calls, 3.045 s of self-time, 36% of the profiled run. Two cautions that the profiler itself teaches: self-time excludes subcalls, and profiling changes the timing (8.44 s here versus 3.71 s unprofiled). Read the share, not the seconds. The share points at parsing.
A profile says which function, not why
So look at the input.
$ python3 count_distinct.py sales.csv 1,000,000 rows, 365 distinct dates
One million calls, 365 distinct strings. The same date string gets parsed about 2,740 times.
The two-line change
from collections import defaultdict
+from functools import lru_cache
+@lru_cache(maxsize=None)
def parse_date(s):
return datetime.strptime(s, "%Y-%m-%d %H:%M:%S")
Prediction before running it: strptime executes 365 times instead of 1,000,000, and the report does not change.
Did the prediction hold?
$ python3 report_cached.py sales.csv report2.txt wrote report2.txt: 1,000,000 rows in 1.20 s $ cmp report.txt report2.txt && echo identical identical $ python3 cache_check.py sales.csv CacheInfo(hits=999635, misses=365, maxsize=None, currsize=365)
365 misses, 999,635 hits. The cache holds exactly 365 entries.
Why it works
strptime is expensive; a dictionary lookup is cheap. With 365 distinct inputs the cache misses 365 times and hits everywhere else. Which raises the real test: what if every date is different?
The limit: when inputs don't repeat
Three data sets, five runs per version, medians. These are whole-process timings (startup and shutdown included), which is why they sit above the single in-script runs at the top. Compare plain and cached within a row, not across the two timers.
$ python3 bench.py --table
distinct dates plain cached speedup same output
365 3.96s 1.43s 2.77x yes
20,000 3.70s 1.49s 2.48x yes
1,000,000 3.76s 3.98s 0.95x yes
1,000,000 rows · 5 runs each, median · Python 3.14.6
Here, caching paid when inputs repeated. With every row unique, the cache cost about 6% and returned nothing.
When to use it
Try caching an expensive pure function when its inputs repeat. Count the inputs before you trust the speedup.
- Use when: the same strings hit a parser thousands of times; the function is pure (same input, same output); you counted, and distinct inputs ≪ calls.
- Doesn't fix: unique inputs (here, about 6% slower); loading — a separate benchmark took about 0.9 s just to read the CSV and build the rows; memory — the cache keeps every distinct key.
Start by profiling your script and inspecting its most expensive functions. Check their inputs, then benchmark any change without the profiler:
python3 -m cProfile -s tottime your_script.py
Reproduce it
Code, data generator and every capture: in the pinned comment under the video
python3 make_data.py --rows 1000000 --unique 365 --out sales.csv python3 report.py sales.csv report.txt python3 -m cProfile -s tottime report.py sales.csv report.txt | head -12 python3 count_distinct.py sales.csv python3 report_cached.py sales.csv report2.txt && cmp report.txt report2.txt && echo identical python3 cache_check.py sales.csv python3 bench.py --table
Two timing scopes appear above on purpose. The 3.71 s and 1.20 s are single measurements timed inside the script. The table reports five-run median whole-process times. They are not the same timer, and the ratios are what transfer.
Comments