Why PDFs Are the Size They Are: Stream Filters, Object Streams and Linearisation

A stream, a dictionary, and two keys

A PDF file is a collection of numbered objects. Anything with bulk — page content, image XObjects, embedded font programs, ICC profiles, XMP metadata, and from PDF 1.5 the cross-reference data itself — is stored as a stream object: a dictionary followed by a run of raw bytes. The dictionary carries a /Length, and optionally a /Filter naming one filter or an array of filters to be applied in order, and a /DecodeParms carrying the parameters each of those filters needs. A consumer reads the dictionary, applies the named decode filters in sequence, and recovers the original bytes.

There is nowhere in the file to record a document-wide compression level, because no such concept exists in the format. This is not a pedantic distinction: it is why two visually identical documents can differ by an order of magnitude, why "reduce file size" commands behave unpredictably across files, and why the question "what compression does this PDF use" usually has several correct answers at once — one per stream.

Borrowed codecs, inherited plumbing

None of the compression in a PDF was designed for PDF. Group 4 coding came from facsimile, where ITU-T T.6 had been in force for years before the format existed. DCTDecode carries the JPEG datastream of ITU-T T.81, JBIG2Decode carries ITU-T T.88, JPXDecode carries JPEG 2000 from ISO/IEC 15444, and FlateDecode carries zlib and DEFLATE from two IETF RFCs. Crediting any single person or company with inventing PDF compression misreads the architecture. The plumbing has an inheritance of its own: PostScript already used named filters, and the ASCII transport encodings in clause 7.4 came across from it directly. What PDF did was carry that filter mechanism into a random-access object format, where every stream declares its own encoding and hands a decoder the parameters it needs, rather than inheriting a setting from the document around it.

Two naming traps follow. FlateDecode is not ZIP — the filter decodes a zlib wrapper around the DEFLATE algorithm, not the PKZIP archive container, whatever product interfaces label it. And DCTDecode does not embed a JPEG file: it embeds a JPEG entropy-coded datastream inside a PDF image XObject, with no JFIF wrapper, and the colour space, bit depth and pixel dimensions read from the PDF dictionary rather than from the image data. The same applies to JPXDecode, which is the JPX structure of ISO/IEC 15444-2 under PDF-imposed restrictions, not a JP2 file pasted into the document.

Version is not a detail

Because filters were added over time, the version recorded in a file determines what its producer was even allowed to do. Any statement of the form "PDFs are compressed by default" is anachronistic without a version attached.

1. Earliest versions

   No FlateDecode. LZWDecode is the general-purpose lossless option, the ASCII filters handle transport, and the cross-reference table is a fixed-width, uncompressed ASCII list.

2. PDF 1.2

   FlateDecode becomes permitted — zlib per RFC 1950 wrapping DEFLATE per RFC 1951. The Library of Congress describes it as the most commonly used method to compress the text content of PDFs.

3. PDF 1.3

   DCTDecode, which had always accepted baseline JPEG, is extended to progressive JPEG.

4. PDF 1.4

   JBIG2Decode arrives for bi-level images at 1 bit per pixel, referencing ISO/IEC 14492:2001 with amendments 1 and 2 and excluding amendment 3.

5. PDF 1.5

   JPXDecode adds JPEG 2000. More consequentially for size, object streams (7.5.7) and cross-reference streams (7.5.8) let the file compress its own object and cross-reference machinery for the first time.

6. PDF 1.7 / ISO 32000-1:2008

   PDF 1.7 is fast-tracked into ISO by TC 171/SC 2 from Adobe's PDF Reference sixth edition, moving the normative text from a vendor reference to an ISO standard.

7. PDF 2.0 / ISO 32000-2

   A revised standard, second edition dated 2020, with its own published errata.

The second layer: compressing the file's own bookkeeping

The classic cross-reference table is a plain-text index: one fixed-width entry per object, each giving a byte offset into the file. It is human-readable, robust, and completely uncompressed, and in a document made of many small objects — tagged structure, annotations, form fields, link destinations, outline entries — the bookkeeping can be a surprisingly large fraction of the total. From PDF 1.5, object streams pack many non-stream objects into a single stream that can itself be Flate-encoded, and cross-reference streams replace the ASCII table with a compressed binary stream. For structure-heavy documents this is often the largest single saving available, and it touches no image at all.

Linearisation is structure, not compression

Linearised PDF is defined in a normative annex — Annex F — and does something orthogonal to encoding. The objects needed to render the first page move to the front of the file and hint tables are added, so a viewer fetching the document over a slow link can display page one from a prefix of the bytes instead of waiting for the trailer to locate the catalogue. The cost is duplication plus the hint tables, which is why a linearised file is usually slightly larger. Precision matters here because product menus place "Fast Web View" next to "Reduce File Size" and write-ups then treat them as one operation: the first is a normative structural property of the format, the second a vendor feature that re-encodes content.

What an exporter actually did to your file

The format permits; the exporter decides. The clearest public record of how an exporter of this lineage behaves is the documentation for Ghostscript's pdfwrite device, which adopted Distiller's parameter names and publishes their per-preset defaults; Adobe's own Acrobat and PDF Library SDK documentation covers the manufacturer side.

Fonts deserve their own caution, because the intuition that subsetting shrinks files is only sometimes true. Embedding a font program (as FontFile, FontFile2 or FontFile3, depending on the font technology) buys fidelity at the cost of bulk. Subsetting removes unused glyphs — but it also defeats reuse across a set of documents, since each file then carries its own partial copy of a face rather than sharing one. Ghostscript's pdfwrite device documents a MaxSubsetPct default of 100 precisely because a subset can end up no smaller than the whole face. Aggressive subsetting of CID-keyed fonts is also a known cause of broken text extraction and search; the archival consequences of that belong with PDF/A rather than here.

Diagnosing a large file

1. 1

   Find out where the bulk actually is

   Any tool that lists objects with their /Filter and /Length answers this directly. The answer is images, embedded fonts, or structure — and which one it is changes every subsequent decision.

2. 2

   Read the version before comparing anything

   A file written at a low compatibility level has no object streams and no cross-reference streams, so much of its bookkeeping is uncompressed by necessity rather than by choice. Comparing it against a modern file compares two different formats.

3. 3

   Separate downsampling from re-encoding

   Reducing pixel dimensions and changing the filter are different operations with different failure modes. A re-encode to a lossy filter degrades the image even at identical dimensions, and successive exports compound that loss.

4. 4

   Match the filter to the content type

   CCITTFaxDecode and JBIG2Decode are defined for 1-bit data; DCTDecode and JPXDecode for continuous tone. Encoding a scanned bitonal page as JPEG, or a photograph as Group 4 fax data, is the common and expensive mistake.

What the format permits versus what your file says

Reading PDF this way converts an unanswerable question into a series of answerable ones. Which streams dominate this file? Which filter is named on each? What version is it, and does it therefore have object streams at all? Were the images downsampled, re-encoded, or both — and at what target resolution and threshold? Was each font embedded whole or subset? Is the file linearised, and was that intentional? Every one of those has a definite answer recorded in the bytes, which is more than can be said for "what compression level was used".

None of those answers live in a preferences pane, and none can be set once for a document. They are properties of individual objects — a stream's encoding, an image's pixel dimensions, a font's embedding, the compatibility level the file was written at, whether it was linearised or not — written by whichever program produced the file. Changing any of them deliberately is the work of a tool that rewrites an existing file, which is the general category a PDF editor or converter belongs to. Knowing what those operations are called in the format turns a single opaque "optimise" button into choices you can make on purpose, and lets you tell afterwards which of them were actually applied.