⏱ The seven-millisecond demo 0:00
The chapter opens with a function that computes the average fare of ten million New York taxi trips. Reading a CSV of almost a gigabyte, it takes under a fifth of a second. Change one line to read a Parquet file with the same trips and two things happen: the file is a sixth of the size, and the function takes seven milliseconds — about twenty-seven times faster.
📊 Rows versus columns 0:51
Two families of file format, and the split is about which unit is contiguous:
| Storage | Formats | What is adjacent on disk |
|---|---|---|
| Row storage | CSV, JSON Lines, Avro | All nineteen values of one trip |
| Columnar storage | Parquet, ORC, Feather (Arrow IPC) | All ten million fares, then all the times |
In the CSV, each line holds one trip and commas separate its values. Because the values have different lengths, the fare is not at a fixed position in the line — so a CSV reader cannot jump to the fares. It reads every character of every line and counts commas. In the Parquet file, the fares sit next to each other, so the reader reads that column and skips the rest.
His second finding is the more useful one: this is not a Python-tooling artefact. Polars has the fastest CSV reader of the three tools tested, DuckDB takes more than three times as long, pandas more than ten times — but even pandas reads the fares from Parquet seven times faster than Polars reads the same column from CSV. The advantage belongs to the format, not the library.
🔤 Types live in the file 2:08
A CSV stores every value as text. A fare of seventeen dollars and seventy cents is four characters — 1, 7, ., 7 — and turning those characters back into a number is parsing, done on every read, by every reader, forever. A CSV also does not record the type of each column, so readers guess from the first values they see: type inference.
He demonstrates the failure mode precisely. PyArrow writes a fee of one dollar as the digit 1, without the decimal point. Polars reads the file, checks the first hundred rows, sees only whole numbers, and decides the column is integers. About eighty thousand rows later it reaches a fee of thirty cents and the read fails.
🧱 Anatomy of a file: PAR1 and the footer 3:08
The layout, in order:
| Bytes | Content |
|---|---|
| First 4 | PAR1 — the magic number |
| Then | The column data, split into row groups, columns and pages |
| Then | The footer: every column's name, type and where its data starts |
| Last 8 | The footer's length, then PAR1 again |
The footer is written last because the writer only knows where each column starts after writing it. A reader therefore starts at the end: read the final eight bytes, learn the footer's length, then read the footer. In his file the footer takes 26 kilobytes of 158 megabytes — about one six-thousandth of the file, and everything the reader needs to plan its work.
The footer is encoded in Thrift, a compact binary serialisation. That is why a Parquet reader can pull every type and offset without parsing a single character of text — the metadata is numbers in a known binary shape.
🗜 Encodings: dictionary, run-length, bit-packing 3:57
The six-times-smaller file is not primarily about compression. It is about encoding — rewriting values in a smaller form that turns back into exactly the same values. In a column, values of the same kind sit together, and many repeat.
The payment-type column is the demonstration. It has only a handful of distinct values, so:
- Dictionary encoding — write each distinct value once, in a short list, then store each row's position in that list.
- Run-length encoding — when the same position repeats many times in a row, store it once with a count.
- Bit-packing — five positions fit in three bits, not sixty-four.
| Representation | Size |
|---|---|
| Plain 64-bit integers | 76 MB |
| After dictionary + RLE + bit-packing | 2.2 MB |
| After Zstandard on top | 1.2 MB |
📄 Pages and page headers 5:08
Encoded values are grouped into pages, and pages are the unit of both compression and skipping. In his example one row group holds about a million payment types: the first page holds the dictionary of five values, and the next fifty-three pages hold the positions. PyArrow packs up to twenty thousand positions into a page by default.
Each page starts with a short header saying how it is encoded and how many bytes it takes. When twenty thousand consecutive payment types are identical, the page holds one run and its data takes thirteen bytes before compression.
🌡 Compression, and the network threshold 5:39
Compression happens per page, so a reader can decompress one page without touching the others. Zstandard takes the payment types from 2.2 MB to 1.2 MB. The defaults differ by tool, which is worth knowing before you compare file sizes across ecosystems:
| Tool | Default Parquet compression |
|---|---|
| Polars | Zstandard (configurable level, default 3) |
| PyArrow | Snappy |
| DuckDB | Snappy |
His measurements: Zstandard writes about as fast as Snappy and produces a file about a fifth smaller. Gzip and Brotli go smaller still, but with PyArrow's defaults Gzip takes almost thirty times as long to write as Zstandard, and Brotli two and a half times as long to read.
🔀 Sorting changes the file size 6:50
Order the rows by pickup time and almost every timestamp is within a few seconds of the one before it — so adjacent values share most of their bytes and compress well. The sorted file is 144 MB; shuffled, the same data is 213 MB.
He also tests the obvious objection: could you just compress the CSV? Yes — Python 3.14 ships Zstandard in the standard library, and the compressed CSV lands at 188 MB, larger than the Parquet file at 158 MB. And the read is far worse: reading the fares from a compressed CSV takes about five times as long as from the plain CSV, because the reader still decompresses every line and parses it. From Parquet the same column takes seven milliseconds.
🎯 Predicate pushdown and row-group statistics 7:43
Columns can be skipped because they are separate. Rows can be skipped too, because Parquet splits rows into row groups and writes, for each column in each row group, the smallest and largest value into the footer.
His file has ten row groups. Ask for the average fare on February 15th: for each row group the footer holds the earliest and latest pickup time, and when February 15th falls outside that range the reader skips the group entirely without reading a byte of it. Passing the filter down to the reader like this is predicate pushdown.
| Data order | Row groups read | Bytes read | Time |
|---|---|---|---|
| Sorted by pickup time | 1 of 10 | — | baseline |
| Shuffled | 10 of 10 | ~20× more | almost 2× longer |
Then the data-quality twist, which is the chapter's second-best moment. In the real New York dataset, nine trips have pickup times in 2002, 2008 and 2009 — published out of order. Because the file keeps the order New York published, two of those nine land in a row group of March trips. The footer then records a range from December 2002 to March 2024 for that group, so the reader cannot skip it and reads a whole row group of March trips to find one day in February.
📦 Row group size is a trade-off 9:10
Row group size is a writer setting. Smaller groups let a reader skip more precisely; larger groups keep the footer small — and the reader reads the whole footer first, before anything else.
| Row group size | Number of groups | Footer size | Consequence |
|---|---|---|---|
| 10,000 rows | 956 | 26 KB → 2.2 MB | Fine skipping, but the footer itself becomes a read |
| Default | 10 | 26 KB | The balanced case he measured with |
| All rows in one group | 1 | minimal | Reads 27 MB to find one day — 55 ms |
Writer defaults differ, and this is where the video has a factual slip worth correcting. PyArrow's default of about a million rows matches his description. DuckDB's default is indeed 122,880 rows. But Polars defaults to 512² — 262,144 rows, roughly twice what he states, so "DuckDB and Polars put about a hundred twenty thousand" is right for one and wrong for the other.
📜 A short history, and two meanings of "version two" 9:57
| Year | Event |
|---|---|
| 2010 | Google describes Dremel, storing records column by column |
| 2013 | Engineers at Twitter and Cloudera build Parquet on that idea for Hadoop; Parquet 1.0 ships in July |
| 2015 | Parquet graduates from the Apache Incubator to a top-level project (27 April 2015) |
| 2017 | Zstandard compression and a page-level index for skipping are added |
The Hadoop context explains the design constraint: Hadoop read from local disks across many machines, disk was the bottleneck, so a reader that touches only the columns it needs reads less from disk. The name comes from parquet floors — small blocks of wood laid in a pattern, like row groups and columns.
PAR1. His file uses format version 2.6 with version-1 pages — the PyArrow defaults.
⚖️ When columnar loses 10:59
The comparison here is cleaner than the usual "Parquet is faster" claim: two embedded databases, one row-oriented (SQLite) and one columnar (DuckDB), each holding a whole database in one file.
| Operation | Row storage (SQLite) | Columnar (DuckDB) |
|---|---|---|
| Fetch one trip by its number | ~6 microseconds | ~29× slower |
| Append one trip | ~0.2 milliseconds | No append — rewrite the file, ~2 seconds |
| Average fare per month | baseline | ~49× faster |
| Average fare per month, single core | baseline | ~9× faster |
| File size | CSV-sized | A sixth of the CSV, smaller than the compressed CSV |
Two structural facts matter as much as the timings. First, PyArrow, Polars and DuckDB cannot append rows to a finished Parquet file — adding one trip means writing the whole file again, and that takes almost two seconds against SQLite's 0.2 milliseconds. Second, DuckDB uses every core while SQLite uses one, and it is still nine times faster on a single core, so the win is architectural rather than a matter of parallelism.
🤖 Why this matters for AI data 12:26
Many of the datasets models learn from are Parquet files. Hugging Face converts its public datasets to Parquet, and the New York taxi data used throughout is published by the city as Parquet too. Big tech uses the format heavily for the same reasons the video measures: smaller files, and reads that touch only what a query needs.
The training-loop implication is worth spelling out, because the video only gestures at it: a data loader that reads Parquet can fetch a column, or a row group, without materialising the rest of the table. On a shuffled CSV, every epoch re-parses every value of every row — including the nineteen columns your model does not use.
🧐 Where the video is imprecise 13:04
- Polars' default row group size is 262,144 rows (512²), not ~120,000. His pairing of "DuckDB and Polars" is correct for DuckDB (122,880) and wrong for Polars. Everything else he says about the trade-off stands.
- "Only five different values" in the payment-type column is a description of his filtered sample. The TLC schema defines six codes (credit card, cash, no charge, dispute, unknown, voided trip); the compression figures are unaffected either way, but the dictionary is larger than five entries in a full-year file.
- All timings are one machine, one dataset, one run. The 27×, 49× and 29× multipliers are the shape of the effect, not its magnitude on your hardware — and DuckDB's 49× partly reflects using every core.
- The 25 Gbit/s threshold is arithmetic from his own measured numbers (91 MB saved against 24 ms of decompression). It is a good way to think, and it moves with your CPU and network — not a constant to memorise.
✅ Claims checked
| Claim | Verdict | Evidence |
|---|---|---|
Parquet files start and end with the magic bytes PAR1 | Verified | Parquet file format specification |
| The footer holds the schema and column offsets, and is written last | Verified | Spec: file metadata at the end, followed by 4-byte footer length and the magic |
| Footer metadata is encoded with Thrift | Verified | Spec: Thrift compact protocol |
| Encodings include dictionary, run-length and bit-packing | Verified | Spec encoding list (plain, RLE/dictionary, delta variants, byte-stream-split) |
| Compression is applied per page | Verified | Pages are independently compressed and decompressed |
| Polars defaults to Zstandard; PyArrow and DuckDB default to Snappy | Verified | Polars `write_parquet(compression='zstd')`; PyArrow and DuckDB default to snappy |
| PyArrow packs about a million rows per row group by default | Verified | Default `row_group_size` of 1,048,576 rows |
| DuckDB and Polars both default to about 120,000 rows per row group | Half wrong | DuckDB: 122,880 ✓. Polars: 512² = 262,144 rows ✗ |
| Data pages default to 1 MB, which is why ~20,000 positions fit a page | Consistent | Polars/PyArrow default `data_page_size` of 1024² bytes |
| Python 3.14 includes Zstandard in the standard library | Verified | `compression.zstd` module added in Python 3.14 |
| Google described Dremel, column-oriented, in 2010 | Verified | Dremel paper, 2010 |
| Parquet was built in 2013 by engineers at Twitter and Cloudera for Hadoop | Verified | Project history; Parquet 1.0 released July 2013 |
| Parquet became a top-level Apache project in 2015 | Verified | ASF announcement, 27 April 2015 |
| Zstandard and the page index were added in 2017 | Verified | Format history: zstd codec and page-level indexing |
| "Version two" means two different things — format version and page version | Verified, and rarely explained this clearly | Format version governs the type set; page version governs page layout and level encoding |
| Format 2.6 with version-1 pages is PyArrow's default | Verified | PyArrow writes format version "2.6" with `data_page_version="1.0"` unless told otherwise |
| Parquet cannot be appended to; adding a row rewrites the file | Verified | Immutable-by-design; all three writers behave this way |
| Hugging Face converts public datasets to Parquet; NYC publishes taxi data as Parquet | Verified | Datasets Server exposes Parquet; TLC trip data page offers Parquet downloads |
| All speed and size multipliers (27×, 49×, 29×, 20×, ~2×) | Single-run measurements | Demonstrations on one machine and one dataset — direction reliable, magnitude not |
| Network threshold of ~25 Gbit/s for compression to pay off | Derived, not universal | Arithmetic from his own 91 MB and 24 ms figures |
Transcript notes. Auto-captions render "Zstandard" correctly but "Parquet" occasionally as "parquet"/"par kit", "PyArrow" as "PyArrow"/"Py Airrow", and "Iceberg" as "Iceberg". The dataset is the NYC Taxi & Limousine Commission trip records, which has nineteen columns in the yellow-taxi schema — matching his "nineteen values" per row.
🔑 Key takeaways
- The format is the lever, not the library: even pandas reading Parquet beats the fastest CSV reader on the same column, so the win belongs to columnar storage itself.
- Columnar wins because analytics reads a few columns of many rows — and loses for point lookups, which is why SQLite still answers one row in microseconds.
- Types are stored, not inferred. That removes an entire class of production failure: the CSV that loads fine until a decimal appears eighty thousand rows in.
- The footer is at the end by necessity, written in Thrift, and it is what makes skipping possible — a reader's first act is to read the last eight bytes.
- Encoding beats compression. Dictionary, run-length and bit-packing take 76 MB to 2.2 MB; Zstandard then takes it to 1.2 MB. And the dictionary alone saves nothing — bit-packing does the work.
- Compression can make reads faster, not just smaller: below about 25 Gbit/s, transferring the 91 MB you saved costs more time than decompressing.
- Sort by the column you filter on. It shrinks the file (144 vs 213 MB) and it enables row-group skipping — the same rows, ordered differently, are a different storage system.
- Nine bad timestamps can defeat a whole row group. Outlier values inflate min/max ranges in the footer, so data cleaning is literally a query optimisation.
- Row group size trades skipping against footer size: 956 small groups grew the footer from 26 KB to 2.2 MB; one giant group made a one-day lookup read 27 MB.
- "Version two" means two independent things — format version (which types you can store) and page version (how a page is laid out). A file can be format 2.6 with version-1 pages.
- Immutability is the price of all of this: no appends, so a single-row write costs a full rewrite — which is exactly why table formats like Iceberg exist on top.
- Check the defaults you inherit: Polars writes Zstandard at 262,144 rows per group; PyArrow and DuckDB write Snappy at 1,048,576 and 122,880 — the same query can hit very different layouts.
🔗 Sources
- The source video — Python In Production, "why .parquet dominates modern data engineering", 13:04, 5 October 2026
- Apache Parquet file format specification — magic bytes, footer, Thrift metadata, encodings, pages, row groups
- Dremel: Interactive Analysis of Web-Scale Datasets (Google, 2010) — the record-shredding and assembly idea Parquet builds on
- PyArrow documentation — writer defaults for row group size, page size, format and page version
- Polars documentation —
write_parquetdefaults: Zstandard, 512² rows per group, 1 MB pages - DuckDB Parquet documentation — reading, writing and defaults
- Hugging Face Datasets Server: Parquet — how public datasets are exposed as Parquet
- NYC TLC trip record data — the dataset used throughout the chapter
- Python 3.14: compression.zstd — Zstandard in the standard library
- The series — Data Engineering for ML and AI, of which this is chapter one