HeadlinesBriefing favicon HeadlinesBriefing.com

Building Multimodal Workflows with a Local LLM

Towards Data Science •
×

The ciya Ollama runtime lets us run the local LLM Gemma 4 on our own machine, allowing image inputs and structured outputs for private data workflows. In this post, we build a three‑stage process that turns a *Finland* trip folder into organized memory records.

First, Python prepares each image, extracting EXIF metadata such as capture time and GPS coordinates. Next, Gemma 4 analyzes the image, returning a structured record that follows the predefined schema: scene summary, captions, place type, activities, objects, signals, mood, uncertainty notes, and confidence. The model is invoked via `ollama.chat` with image and text inputs and a JSON schema format.

Finally, the individual records are fed back to Gemma 4 to synthesize a trip memory. The output includes a narrative summary, inferred interests, recurring themes, and memorable moments, each linked to specific photo IDs.

A workaround for older Gemma 4 E4B models involves loading two GGUF files (projector and quantized model) and creating a Modelfile. This ensures the model consumes visual content correctly.