PYTHON • WEBHOOKS • DATA ENGINEERING

The failure was not "an AI problem." It was a data-boundary problem: a large webhook body, deeply nested records, Base64-encoded documents, inconsistent dates and malformed input arriving from systems I did not control.

By Unwired Web Solutions | Updated September 22, 2026

Everybody wants to talk about the orchestration layer. Nobody wants to talk about the moment a multi-megabyte payload arrives as a Base64 string buried inside nested JSON and blocks a webhook worker in the middle of the night.

That was the useful lesson from a vendor-document pipeline I worked on at Unwired Web Solutions. The first parser looked fine against a small, clean fixture. Production introduced larger files, inconsistent fields and malformed encodings. Tracing the failure to its actual root cause and rebuilding the parser properly took most of a day — not the quick patch the first look at the traceback suggested. The system did not need a cleverer prompt. It needed explicit boundaries, bounded memory, reproducible tests and a failure path that preserved the rest of the batch.

This build log explains the pattern I used. The code is intentionally generic; adapt the JSON path, limits, MIME allowlist, storage layer and retry policy to your own sender contract.

The short version

  • Network chunks are not JSON records. A streaming HTTP iterator may split one JSON token across several chunks—or combine several records into one chunk.
  • Base64 increases the transmitted representation by roughly one-third before JSON and HTTP overhead, so size limits must account for encoded and decoded data.
  • Padding repair is a compatibility policy, not proof that the payload is correct. Strict validation, file-signature checks and audit metadata still matter.
  • Streaming the JSON document controls document-level memory, but decoding a single Base64 field can still allocate the full file. Put a hard cap on each document or use a true streaming decoder/object-storage path for large files.
  • Bad records should be quarantined with safe metadata. Do not write raw Base64, credentials or document contents to application logs.

Why do Base64 and nested JSON break naive parsers?

Quick answer: Do not call json.loads() on arbitrary network chunks and do not assume every Base64 string is valid. Parse the documented JSON structure incrementally, extract one record at a time, enforce encoded and decoded size limits, repair missing padding only when the sender contract permits it, decode with validate=True, verify the resulting file type, and quarantine bad records without logging their contents.

A typical payload looked conceptually like this:

{
  "vendor_id": "supplier-42",
  "records": [
    {
      "document": {
        "name": "certificate.pdf",
        "content_type": "application/pdf",
        "document_payload": "JVBERi0xLjQKJ..."
      },
      "attributes": {"issued_at": "2026-03-14T08:31:22Z"}
    }
  ]
}

Three different problems were hiding in that one envelope.

1. The code confused transport chunks with complete JSON

Iterating over raw_stream does not guarantee that each item is a complete JSON object. Chunk boundaries are chosen by the server, framework and network stack—not by the JSON grammar. Calling json.loads(chunk) will eventually fail on a split string, split escape sequence or partial object.

The fix is to use the framing the sender actually provides:

  • One normal JSON document: use an incremental parser such as ijson and target the array path, for example records.item.
  • Newline-delimited JSON (NDJSON/JSON Lines): read complete lines, then parse each line.
  • A queue or multipart protocol: let that protocol define record and file boundaries.

2. Base64 padding errors crashed the worker

Standard Base64 uses four-character groups and may end with = padding. Some integrations omit that padding. Python raises binascii.Error when strict decoding sees incorrect padding or non-alphabet characters.

Missing padding can sometimes be reconstructed when the sender contract explicitly allows unpadded Base64. A length remainder of 2 needs two = characters; a remainder of 3 needs one. A remainder of 1 is structurally impossible to repair safely and should be rejected.

3. The pipeline had no useful memory boundary

Parsing the entire request into a Python object retains the JSON text, nested structures and large encoded strings at the same time. Decoding then creates another bytes object. The peak can be much larger than the final file. Garbage collection may add pressure, but object retention and duplicate allocations are the primary design issue.

Incremental parsing limits how much of the outer document is materialized. Per-file size caps limit each decode. If a single document can exceed that cap, the design should change: use direct object uploads, multipart transport, signed upload URLs or a decoder that writes aligned Base64 blocks to temporary/object storage.

The corrected Python pattern

The example below assumes one JSON document with an array at records. It streams one record at a time, reads the nested document safely, applies a strict size policy and isolates failures. It does not pretend that arbitrary HTTP chunks are complete JSON.

from __future__ import annotations

import base64
import binascii
import hashlib
import logging
from dataclasses import dataclass
from typing import Any, BinaryIO, Iterator

import ijson

logger = logging.getLogger("uws.pipeline")
MAX_ENCODED_BYTES = 20 * 1024 * 1024  # set from your contract
MAX_DECODED_BYTES = 15 * 1024 * 1024
ALLOWED_TYPES = {"application/pdf", "image/tiff"}


class PayloadError(ValueError):
    pass


@dataclass(frozen=True)
class DecodedDocument:
    name: str
    content_type: str
    data: bytes
    sha256: str


def required_path(value: Any, *path: str) -> Any:
    current = value
    for key in path:
        if not isinstance(current, dict) or key not in current:
            raise PayloadError(f"Missing required field: {'.'.join(path)}")
        current = current[key]
    return current


def decode_base64_document(record: dict[str, Any]) -> DecodedDocument:
    name = required_path(record, "document", "name")
    content_type = required_path(record, "document", "content_type")
    raw = required_path(record, "document", "document_payload")

    if content_type not in ALLOWED_TYPES:
        raise PayloadError("Unsupported declared content type")
    if not isinstance(raw, str):
        raise PayloadError("document_payload must be an ASCII string")

    try:
        encoded = raw.encode("ascii")
    except UnicodeEncodeError as exc:
        raise PayloadError("Base64 input contains non-ASCII characters") from exc

    if len(encoded) > MAX_ENCODED_BYTES:
        raise PayloadError("Encoded document exceeds the size limit")

    remainder = len(encoded) % 4
    if remainder == 1:
        raise PayloadError("Invalid Base64 length")
    if remainder:
        encoded += b"=" * (4 - remainder)  # only if sender policy allows

    try:
        data = base64.b64decode(encoded, validate=True)
    except (binascii.Error, ValueError) as exc:
        raise PayloadError("Invalid Base64 payload") from exc

    if len(data) > MAX_DECODED_BYTES:
        raise PayloadError("Decoded document exceeds the size limit")

    return DecodedDocument(
        name=name,
        content_type=content_type,
        data=data,
        sha256=hashlib.sha256(data).hexdigest(),
    )


def iter_results(stream: BinaryIO) -> Iterator[tuple[str, Any]]:
    for index, record in enumerate(ijson.items(stream, "records.item")):
        try:
            yield "ok", decode_base64_document(record)
        except PayloadError as exc:
            # Log safe metadata only; never the document payload.
            logger.warning("Rejected record index=%s reason=%s", index, exc)
            yield "error", {"record_index": index, "reason": str(exc)}

Important limitation: ijson keeps the outer document incremental, but the code above still materializes each record and decoded file. That is appropriate only when MAX_ENCODED_BYTES and MAX_DECODED_BYTES fit comfortably within the worker budget. For genuinely large files, change the transport or decode aligned Base64 blocks directly to storage.

Tested with: Python 3.11.15 and ijson 3.5.1, against six redacted fixtures — valid padded Base64, valid unpadded Base64 (padding repaired), an impossible length remainder, a disallowed content type, a missing required field and a non-ASCII payload. The two valid records decoded correctly with matching content hashes; all four invalid records were rejected and quarantined with the correct reason code.

What the production decode chain should do

  • Authenticate before heavy work. Verify the webhook signature or mTLS identity, enforce a request-size limit at the edge and reject stale/replayed requests.
  • Parse according to the documented framing. Use ijson for a JSON array, line parsing for NDJSON, or the queue/multipart API's own record boundaries.
  • Validate the envelope and nested schema. Check required keys and types before allocating a decode buffer. Normalize date formats in a separate step.
  • Apply encoded-size and decoded-size limits. Reject before decode when possible; remember that malformed data and decompression-style amplification can exhaust memory.
  • Decode strictly. Use validate=True. Repair padding only when the integration contract allows omitted padding; record that a repair occurred.
  • Inspect the bytes. Do not trust a declared MIME type or filename extension. Check the file signature with a maintained library, then scan or sandbox files as required.
  • Persist idempotently. Store a sender event ID and/or content hash so retries do not create duplicate records or files.
  • Quarantine failures. Preserve a safe event identifier, record index, reason code, sender and timestamp—never the document body in ordinary logs.
  • Observe the pipeline. Track accepted, repaired, rejected, oversized, duplicate and processing-latency counts by sender and schema version.

The tests that matter more than the prompt

A clean 50 KB fixture proves very little. Build a table-driven suite that includes:

  • Valid padded and permitted unpadded Base64.
  • Invalid alphabet characters, whitespace-policy violations and impossible length remainder 1.
  • Empty files, boundary-size files, one byte over each limit and very large nested metadata.
  • JSON split at every awkward boundary: inside strings, escapes, Unicode sequences and array delimiters.
  • Missing keys, wrong types, nulls, duplicate event IDs and mixed ISO-8601/epoch date inputs.
  • Declared PDF with non-PDF bytes; suspicious filenames; files that fail malware/content inspection.
  • One bad record between two good records to verify batch isolation.
  • Retries, worker restarts and storage/database failures to verify idempotency and cleanup.

Measure peak resident memory, not just average CPU. A pipeline can pass functional tests and still fail under concurrent large records.

Where Claude Code helped—and where it did not

Claude Code was useful as an interactive development partner: drafting scaffolding, generating test cases, comparing exception-handling options and accelerating iteration. It was not the source of truth for payload framing, memory limits, security policy or acceptance criteria.

Those controls had to come from the sender contract, real payload samples, the runtime budget and manual review. That is the broader lesson: generated code can shorten the path to a prototype, but it does not remove the need to understand the bytes crossing a trust boundary.

The unglamorous rule for AI-built integrations

Prompt-only automation is brittle when it sits between systems you do not control. The failure modes are not philosophical. They are specific: a field changes type, a body exceeds the proxy limit, a retry creates a duplicate, a malformed file reaches storage, or a log line leaks the payload.

Real automation is not defined by how quickly code appears. It is defined by what happens when the outside world sends bad data. A production pipeline should reject precisely, recover predictably and leave enough evidence to diagnose the failure without exposing the underlying document.

Frequently asked questions

Why does Base64 fail with "Incorrect padding" in Python?

Standard Base64 is grouped in four-character blocks and may require = padding. If an input is truncated or a sender omits permitted padding, base64.b64decode can raise binascii.Error. Only repair omitted padding when the sender contract allows it; reject impossible lengths and decode with validate=True.

Can I parse each chunk from an HTTP stream with json.loads()?

No. HTTP or framework chunks are transport fragments, not guaranteed JSON records. Use an incremental parser for one JSON document, read full lines for NDJSON, or use the boundaries defined by the transport protocol.

Does streaming JSON eliminate memory problems?

It reduces the memory used by the outer document, but it does not automatically stream a Base64 string or the decoded file. A per-record parser may still hold the entire encoded field and decoded bytes. Enforce file limits or move large documents to a storage-oriented upload path.

Should I automatically add missing Base64 padding?

Only as an explicit compatibility rule for a known sender. Record when padding is repaired, reject a length remainder of 1, and keep strict alphabet validation. Padding repair cannot prove that the resulting file is authentic or complete.

How do I validate a Base64 file after decoding?

Check the decoded size, calculate a content hash, inspect the file signature with a maintained library, enforce an allowlist, and run the security scanning required by your environment. Do not rely on a filename or declared MIME type alone.

Technical references

Need to harden an AI workflow?

If a critical process depends on fragile webhooks, unbounded payloads or generated scripts that have only seen clean test data, an AI Workflow Audit can map the failure points before they become production incidents.

Book the AI Workflow Audit → https://davidhenderson.ca/contact/

Subscribe to the newsletter for more build logs like this one.