How we built a pipeline that preserves context across 100k+ token books using Python, FastAPI, and Claude's context window.

The Problem: Books Don't Fit in a Prompt

When we started building LectuLibre, our AI-powered book translation service, we assumed we could just send an entire book to a large language model and get a translation back. After all, Claude 3.5 Sonnet advertises a 200k token context window. But a typical novel is 80,000–120,000 words, which is roughly 100,000–150,000 tokens. That's technically within limits, but...

Cost: translating a 100k token book in one shot is expensive.
Quality: LLMs lose attention to early chapters when processing long contexts.
Rate limits and timeouts.
Hard to resume if fails.

So we needed a chunking strategy that preserves cross-chapter context: terminology, character voice, consistent style.

Our Approach: Sliding Window Chunks + Context Summary

We split the book into overlapping chunks, translate each with a context buffer containing:

A running glossary of terms and character names
A summary of previous chapters
The current chunk's raw text

The pipeline:

Parse EPUB/PDF into plain text with chapter metadata.
Split text into token-aware chunks with overlap.
For each chunk, fetch glossary + previous summary from PostgreSQL.
Call LLM to translate with system prompt.
Extract new terms from translation, update glossary.
Generate a concise summary of the translated chunk, store for next chunk.
Assemble final translated book.

Code: Token-Aware Chunking

We tried LangChain's RecursiveCharacterTextSplitter first, but it works on characters, not tokens, so some chunks exceeded the model's token limit. We switched to tiktoken for exact token counting.

import tiktoken
from typing import List, Tuple

def chunk_text_by_tokens(
    text: str,
    target_tokens: int = 3000,
    overlap_tokens: int = 500,
    model: str = "claude-3-5-sonnet-20240620"
) -> List[str]:
    # Use cl100k_base for Claude? Claude uses its own tokenizer but tiktoken's
    # cl100k_base approximates well enough for chunk sizing.
    enc = tiktoken.get_encoding("cl100k_base")
    tokens = enc.encode(text)

    chunks = []
    start = 0
    while start < len(tokens):
        end = min(start + target_tokens, len(tokens))
        chunk_tokens = tokens[start:end]
        chunks.append(enc.decode(chunk_tokens))

        if end == len(tokens):
            break
        # Move start back by overlap, but not before current start
        start = max(start + target_tokens - overlap_tokens, start + 1)

    return chunks

But books have paragraphs/chapters; we don't want chunks to split mid-sentence. We later added a paragraph-aware post-processing that adjusts boundaries to the nearest paragraph break within the token window. That improved translation quality.

Translation Pipeline with Context

We use Anthropic's Python SDK for Claude, and DeepSeek as a cheaper alternative for simpler passages. Here's the core translation function:

import anthropic
from tenacity import retry, stop_after_attempt, wait_exponential

client = anthropic.Anthropic(api_key="...")

SYSTEM_PROMPT_TEMPLATE = """You are a professional literary translator.
Translate the following text from {source_lang} to {target_lang}.
Preserve the author's style, tone, and voice.

Context from previous chapters:
{chapter_summary}

Glossary of established terms and names (use exactly these translations):
{glossary}

Translate only the provided text. Do not add explanations.
"""

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def translate_chunk(
    chunk_text: str,
    source_lang: str,
    target_lang: str,
    glossary: dict,
    chapter_summary: str,
    model: str = "claude-3-5-sonnet-20240620"
) -> str:
    glossary_str = "\n".join([f"{k} -> {v}" for k, v in glossary.items()])
    prompt = SYSTEM_PROMPT_TEMPLATE.format(
        source_lang=source_lang,
        target_lang=target_lang,
        chapter_summary=chapter_summary,
        glossary=glossary_str
    )

    response = client.messages.create(
        model=model,
        max_tokens=4096,
        temperature=0.3,
        system=prompt,
        messages=[{"role": "user", "content": chunk_text}]
    )
    return response.content[0].text

We use tenacity for retries due to occasional API timeouts. The glossary is a simple Python dict stored as JSONB in PostgreSQL via SQLAlchemy. We update it after each chunk:

def update_glossary(session, glossary, source_chunk, translated_chunk):
    # Simplified extraction: ask LLM to list new terms
    extraction_prompt = f"Extract proper nouns, technical terms, and repeated phrases from this translated text. Return a JSON list of objects with 'source' and 'translation'."
    # ... call Claude or DeepSeek ...
    new_terms = extract_terms(translated_chunk)
    for term in new_terms:
        glossary[term['source']] = term['translation']
    session.execute(
        text("UPDATE books SET glossary = :glossary WHERE id = :book_id"),
        {"glossary": json.dumps(glossary), "book_id": book_id}
    )

We maintain a chapter summary by asking the LLM to summarize the translated chunk in 2-3 sentences, then concatenate with previous summary but keep it under 500 tokens.

Results and Trade-offs

For a 120,000-word novel (≈150k tokens), our pipeline creates about 60 chunks of 3000 tokens with 500 overlap. Translation using Claude 3.5 Sonnet costs around $12–15 per book, and takes about 20–30 minutes. Before chunking, a single-shot translation often produced inconsistencies (e.g., a character's name changed midway) and occasionally timed out. Chunking reduced errors significantly; we measured a 40% drop in user-reported translation errors.

But it wasn't perfect. Early on we set overlap to 100 tokens (about 3%), and translators noticed context breaks at chunk boundaries: a sentence referencing "the previous chapter's event" would be mistranslated because the LLM didn't have that context. We increased overlap to 500 tokens (16-17% of chunk size), which helped.

Another failure: we initially included the full glossary plus all previous summaries in every prompt, which bloated the context and sometimes confused the model. We now cap the glossary to the 50 most recent terms and keep the summary under 500 tokens.

Lessons Learned

Token-aware chunking is essential. Don't rely on character counts; use tiktoken or the model's tokenizer.
Overlap matters more than you think. Start with 15-20% overlap and test boundaries.
Maintain a running glossary, but prune it. Too much context can hurt as much as too little.
Use summarization to bridge long-range context. A short summary of prior chapters is often more useful than raw text.
Choose your model per chunk. We use Claude for literary quality and DeepSeek for more mechanical sections (table of contents, footnotes) to save cost.

What's Next?

We're exploring hierarchical translation: first translate chapter summaries, then use those as context for translating full chapters. That might improve coherence for very long books with complex plots.

Takeaway: Translating entire books with LLMs is feasible if you treat it as a context-management problem, not a single-prompt problem. Chunking with overlap, glossary, and summary bridges the gap between the model's context window and a book's length.

Open question for the community: How do you handle idiomatic expressions or cultural references that need adaptation rather than literal translation? Any techniques beyond a glossary?

StackFeed

Translating Full Books with LLMs: Our Chunking Strategy for Long-Form Context