AI Content Intelligence Pipeline

This public project is a sanitized implementation based on architectural patterns used in a larger client content-automation system. It preserves the interesting engineering ideas without exposing client-specific prompts, data, private integrations, or production infrastructure.

The repository is an NDA-safe reference implementation, not the complete production client system. It demonstrates workflow structure, data-processing patterns, AI integration boundaries, schema-driven output, and deterministic orchestration.

Source content can be missing fields, differently formatted, incomplete, noisy, or variable across sources. LLM output introduces another source of variability. A single prompt from raw input to final publication makes validation, debugging, failure isolation, and output consistency harder.

The engineering question is how to introduce AI without surrendering control of the workflow. The answer is explicit stages with contracts between them.

The reference implementation reads an example RSS feed through a pluggable ingestion layer. The adapter turns feed entries into RawItem[], keeping source acquisition separate from downstream processing.

The reference uses example RSS ingestion. The architecture pattern is a replaceable ingestion boundary.

Raw source items become consistent internal records before later stages operate. In the verified schema, a RawItem with a feed link becomes a NormalizedItem with a normalized URL, published value, text, and tags.

Normalize variability before asking the model to interpret it.

{
  "title": "Indie trio drops a new alternative single",
  "url": "https://example.com/indie-single",
  "published": "2026-02-05",
  "text": "An indie band released a new alternative track...",
  "tags": ["indie", "electronic"]
}

{
  "title": "Indie trio drops a new alternative single",
  "url": "https://example.com/indie-single",
  "one_sentence": "An indie band released a new alternative track...",
  "bullets": ["The song leans into an underground aesthetic..."],
  "tags": ["indie", "electronic"]
}

The sanitized implementation applies basic tag rules as a placeholder for more advanced classification logic. It checks normalized title and text against explicit keyword rules, assigns matching tags, and keeps only items that satisfy the example policy.

Not every pipeline decision belongs to an LLM. Rule-based filtering remains deterministic when the requirement is explicit.

The summarization stage is defined behind a stable interface and produces structured JSON summaries. Pydantic models describe the summary boundary with a title, URL, one-sentence summary, bullets, and tags. Schema validation constrains what downstream stages are allowed to accept.

The model contributes interpretation inside a defined contract; it does not decide the pipeline sequence or publishing format.

Mock mode lets local development exercise orchestration without API calls. Stage integration can be inspected independently of model availability, deterministic sample output makes behavior easier to reason about, and the summarizer implementation can be swapped without fusing AI availability to the pipeline runner.

USE_MOCK_SUMMARIZER=true

Because normalization, classification, summarization, and publication are distinct stages, engineers can inspect where output changed. Instead of debugging raw source directly against a mysterious final result, the reasoning path is visible from raw input to normalized representation, classified set, structured summary, and final post.

Stage boundaries turn an opaque AI workflow into an inspectable software system.

When final output is incorrect, the pipeline structure helps distinguish an ingestion problem from a normalization problem, a classification problem, an AI interpretation problem, or a publishing problem. The architecture makes those questions local instead of treating the whole workflow as one opaque operation.

Models handle interpretation. Code controls the workflow.

AI output becomes useful in a software system when downstream code receives known structure rather than free-form prose. Pydantic schemas, explicit serialization, JSON intermediate artifacts, and Markdown transformation make the next operation predictable.

Downstream code should not parse vibes.

• Python 3.11+
• Pydantic
• feedparser
• python-dotenv
• OpenAI-compatible integration

The useful engineering work in an AI pipeline is not only the model call. It is the structure around it: controlled inputs, normalized representations, explicit stage boundaries, validated outputs, deterministic orchestration, and inspectable artifacts.

Reliable AI systems are built from controlled transitions, not single prompts.