Project Summary

A multi-year progression, starting in 2022, from early generative-AI experimentation into today’s agentic AI architecture, developed while the technology, tooling, and industry practices themselves were still rapidly evolving. Early work included a voice-interactive AI pipeline (speech-to-text, a language model, text-to-speech) built before ChatGPT introduced native voice conversations in September 2023. A later, deliberate decision to build on retrieval-augmented generation rather than fine-tuned models avoided an estimated 5x ongoing cost premium for the evaluated workload. The current phase centers on agentic architecture, Databricks Genie, and a genuine build-versus-buy evaluation of vendor tooling, deliberately limiting provider sprawl to hold down integration and operating complexity, accelerated by how much MCP has reduced the integration work needed between AI applications, tools, and data sources.

Problem

The initial problem wasn’t a single application requirement; it was determining whether rapidly emerging generative AI capabilities were mature enough to justify real organizational investment, while building enough hands-on expertise, starting in 2022, to tell practical use cases apart from hype. There was no established organizational practice to build from yet. That judgment had to be developed through direct experimentation while industry practices were still immature.

As the technology matured, the questions got more concrete: how to ground models safely on organizational knowledge, when retrieval was the better call versus customizing a model directly, and how to connect increasingly capable models to enterprise tools without the integration layer becoming ungovernable. The problem never really closed; it shifted shape as the technology did.

Constraints

• No existing internal AI practice, tooling, or playbook to build from; this had to be learned in parallel with everyone else in the industry.
• Real hype-versus-capability confusion among stakeholders had to be addressed directly and honestly, not just technically worked around.
• Early experimentation with organizational data was deliberately limited to public and otherwise non-sensitive content while security, privacy, and governance patterns were still being established.
• In the evaluated workload, fine-tuned models were projected at roughly 5x the ongoing operating cost of a comparable retrieval-augmented approach, a real, recurring cost constraint on architecture choices, not a one-time budget line.
• The underlying technology landscape kept changing fast enough that architecture decisions had to be revisited periodically rather than treated as settled.

Architecture

Early Experimentation: Building a Voice-Interactive AI Pipeline

The earliest work was genuinely exploratory: informal “AI labs” testing what GPT-era models could actually do. One outcome from that period was a command-line interface that took spoken input, transcribed it with Whisper, submitted the transcription to the AI model, and used Azure’s speech services to convert the response back into voice for the user, a full voice-in, voice-out interaction loop, stitched together from separate components before ChatGPT introduced native voice conversations in September 2023.

Grounding on Internal Data, and an Early Chatbot

The next step was grounding models on the organization’s own non-sensitive data, starting with public website content, indexed and retrieved at query time so responses could reflect organizational context without retraining the underlying model, to build a chatbot that answered in that context rather than staying purely generic. This is also where the real, hands-on lessons about model tuning and retrieval-augmented generation started: RAG wasn’t an abstract technique yet, it was the thing that made a model’s answers actually reflect what the organization needed it to know.

Choosing RAG Over Fine-Tuning on the Economics

As the technology matured, fine-tuning became a real architectural option, and a real cost question. For the workload being evaluated, fine-tuned models were projected at roughly 5 times the ongoing cost of a comparable retrieval-augmented setup using vector search and embeddings. That required translating the technical tradeoff into a business decision: whether the incremental capability justified approximately five times the projected ongoing operating cost. Retrieval-augmented generation was selected because it met the workload’s functional and interactive requirements without that additional operating-cost premium.

The Current Phase: Agentic Architecture and MCP-Driven Interoperability

The most recent shift has been toward agentic systems: models that don’t just generate a response but can invoke tools, retrieve external context, maintain state across steps, and execute multi-step tasks under defined controls, rather than answering once and stopping. The Model Context Protocol has made that architecture meaningfully more practical: it standardizes how AI applications connect to external tools and data sources, cutting down the bespoke integration work that used to be required for every application-to-tool pairing. Databricks has recently entered this picture too, specifically around Genie’s conversational data-exploration capability, not as a general multi-provider model-serving expansion, but a targeted addition where it fits.

That’s brought back a very traditional infrastructure question in a new context: build the agentic tooling in-house, or evaluate and adopt vendor products, the same build-versus-buy tradeoff that applies to any other platform decision, now being worked through for AI specifically. The scope has also been deliberately kept narrow: the current architecture limits the number of model and platform providers in active use, Azure-hosted models and Databricks Genie, not a wider roster of external providers, a conscious choice to hold down integration, governance, and operating complexity while the agentic architecture is still evolving, not an oversight.

Engineering Challenges

Separating Real Capability From Hype

A recurring, ongoing part of this work has been telling people honestly what the technology could and couldn’t actually do, for stakeholders making decisions based on what they’d read or heard, not just for engineers building with it. That’s a harder problem than it sounds: overselling capability erodes trust the first time it fails to deliver, and underselling it means missing real opportunities the technology genuinely supports.

Building Organizational Capability, Not Just a System

A meaningful part of this work was enablement: helping people get started with AI tools, understand what the models could and couldn’t reliably do, structure instructions that actually got useful results, and critically evaluate what came back rather than trusting it by default. That’s a different kind of engineering work than building infrastructure: it’s building capability in other people, which doesn’t show up in an architecture diagram but was as real a part of the job as anything that does.

Justifying Architecture on Cost, Not Just Capability

Choosing retrieval-augmented generation over fine-tuning wasn’t purely a technical call; it required making the cost tradeoff legible to the business and getting a real answer about what they were willing to pay for a marginal capability improvement. Architecture decisions that only make sense on a whiteboard don’t hold up if nobody signs off on what they cost to run.

Re-Architecting as the Field Itself Moves

Because the underlying technology changed so quickly, this was never a “build it once” problem. Each phase, early experimentation, RAG-based grounding, and now agentic architecture, represented a real redesign, not an incremental patch on what came before, because the capabilities and economics underneath kept shifting enough to justify starting the architecture conversation over.

Results

• Built a working voice-interactive AI pipeline (speech recognition, language model, speech synthesis) by composing separate components, before ChatGPT introduced native voice conversations in September 2023.
• Established retrieval-augmented generation, with vector search and embeddings, as the preferred grounding pattern over fine-tuning for the evaluated workload, avoiding an estimated 5x ongoing operating-cost premium while still meeting functional and interactive requirements.
• Limited early organizational-data experiments to public and non-sensitive content, establishing a conservative security boundary while governance patterns were still being established.
• Built organizational AI literacy directly, through hands-on enablement around model limitations, effective instruction, and critical evaluation of outputs, not just technical infrastructure.
• Currently evaluating and redesigning around agentic architecture, with MCP substantially reducing the bespoke integration work needed between AI applications, tools, and data sources, and Databricks Genie entering the picture for conversational data exploration specifically.
• Maintained a build-versus-buy discipline for AI tooling consistent with how other enterprise infrastructure decisions get made, rather than treating AI as a special case exempt from that evaluation.
• Deliberately limited provider sprawl during the current phase, keeping the architecture focused on Azure-hosted and Databricks capabilities already in use rather than integrating every available external model platform.

Technologies

Models & Interaction

Azure OpenAI / GPT, prompt engineering, fine-tuning, retrieval-augmented generation (RAG), vector search and embeddings

Voice & Multimodal

Whisper (speech-to-text), Azure Speech services (text-to-speech)

Agentic & Interoperability

Model Context Protocol (MCP), agentic AI architecture, Databricks Genie