tfpdev
01 / 09Part 1 · Chunking

Where to Cut

Before an AI can look anything up, every file gets cut into small pieces. Cut them evenly and a price gets separated from its label. Cut them by section and each piece keeps its name and date.

01Step 1

Try it

One file, cut into pieces two ways. These are the real pieces the bot searches. Pick a fact and see whether its piece still says where it came from.

Pick a file
Equal pieces…
    By section…

      These are the real pieces the bot searches. Grey text at the start of a piece is repeated from the one before, so nothing falls in the gap.

      02Step 2

      What actually mattered

      The AI only sees the piece

      The AI doesn't read the whole file, only the pieces it's handed. If a price arrives without its year, it has no way to know it's out of date.

      Cut where the headings are

      Files already come in sections. Cutting there keeps tables whole and sentences in one piece.

      Put the name on every piece

      A few extra words per piece: the file's name, its date, and whether it's out of date. That's what lets the AI pick the current price over the old one.

      03Step 3

      The code

      What the two versions do, in one sentence. The code itself is one click away.

      The quick version slices every file into equal pieces of 400 characters. The careful version cuts at each heading and writes the file's name and date at the top of every piece.

      Show the code anyway
      BreaksBreaks
      Fixed windows with overlap.
      lib/rag/chunk.ts · fixedChunksts
      export function fixedChunks(doc: Doc): Chunk[] {  const text = doc.body.replace(/\s+/g, " ");  const out: Chunk[] = [];  for (let start = 0, ord = 0; start < text.length; start += 400 - 50, ord++) {    const piece = text.slice(start, start + 400).trim();    out.push({ strategy: "fixed", docSlug: doc.slug, ord, heading: "", text: piece });    if (start + 400 >= text.length) break;  }  return out;}
      The usual first attempt. Fast, simple, and blind to structure: a table row lands in one window and its header in the previous one, and no window knows which file or year it came from.
      HoldsHolds
      One labelled chunk per section, running behind the bot.
      lib/rag/chunk.ts · sectionChunksts
      export function sectionChunks(doc: Doc): Chunk[] {  const parts = doc.body.split(/^## /m);  const intro = parts[0].replace(/^# .*$/m, "").trim();  const sections = parts.slice(1).map((p) => {    const nl = p.indexOf("\n");    return { h: p.slice(0, nl).trim(), body: p.slice(nl + 1).trim() };  });  if (intro) sections.unshift({ h: "Overview", body: intro });  const label = doc.status === "archived"    ? `${doc.title} [ARCHIVED, updated ${doc.updated}]`    : `${doc.title} [updated ${doc.updated}]`;  return sections.map((s, ord) => ({    strategy: "section", docSlug: doc.slug, ord,    heading: `${label} › ${s.h}`, text: s.body,  }));}
      One chunk per “## ” section, tables kept whole. Every chunk carries a label with the file's title, date and ARCHIVED flag, and the label is searched and sent to the model with the text.
      04Step 4

      Still curious?

      A few more questions people ask. Open what you need.

      Why not just make the pieces bigger?

      Bigger pieces split fewer tables but make search fuzzier. Smaller ones lose the label. No single size fits every file. The file's own headings usually do.

      What if a file has no headings?

      Then the whole file is one piece, which is fine for an email. A long file without headings gets split by paragraph, still with its name on every piece.

      Why repeat text between pieces?

      So a sentence cut at the edge appears whole in at least one piece. It doesn't fix a price that's far from its heading.