Ziroh Labs’ Hrishikesh Dewan on Kompact AI, CPU-Native LLM Inference, and Making Enterprise AI 70–80% More Cost-Efficient

StartupTalky presents Recap'25, a series of exclusive interviews where we connect with founders and industry leaders to reflect on their journey in 2025 and discuss their vision for the future.

In this edition of Recap’25, StartupTalky speaks with Hrishikesh Dewan, Co-founder & CEO of Ziroh Labs, who shares how the company is rethinking AI infrastructure by enabling production-grade large language models to run efficiently on CPU architectures, eliminating the traditional dependency on GPUs. Dewan explains how Kompact AI, Ziroh Labs’ fast-inference AI platform, is designed to optimize token throughput, reduce latency, and maintain model accuracy while delivering an OpenAI-compatible deployment layer for enterprises and developers.

He also highlights the key milestones that shaped 2025, including early benchmark validation with IIT Madras, expansion of support to 300+ LLMs across multimodal workloads, and the launch of Kompact AI One, a semantic caching module built to improve throughput for contextually similar queries. The conversation further explores why CPU-native inference can unlock more predictable cost structures and stronger data control for regulated sectors such as BFSI and healthcare, along with Ziroh Labs’ 2026 roadmap focused on deeper model support, partnerships with OEMs and system integrators, and developer SDKs aimed at accelerating CPU-based AI adoption globally.

StartupTalky: Can you give us a brief introduction to Ziroh Labs and Kompact AI for our readers?

Hrishikesh Dewan: Kompact AI is a fast-inference AI platform that enables AI models to run efficiently on CPU architectures, eliminating the need for GPUs. It focuses on optimising token generation throughput and reducing system latency while maintaining model accuracy. Kompact AI can be embedded in various computational environments and offers an OpenAI-compatible API layer for easy deployment.

The platform has four key components:

• Runtime for CPU execution: Executes open-source or fine-tuned models on CPUs.
• Remote REST-based server: Hosted via NGINX, it provides pluggable modules for custom logic like access restriction and rate-limiting.
• Observability: Implements OpenTelemetry for monitoring various metrics like input/output, SLA, CPU/memory utilisation, and network activity.
• OpenAI-compatible libraries: Allows remote access to models using OpenAI-compatible or native HTTP libraries.

StartupTalky: What inspired you to build a platform that enables LLM inference on CPUs, and how is it different from existing GPU-based solutions?

Hrishikesh Dewan: Today, GPUs remain the primary infrastructure for building and deploying AI systems. While powerful, they are expensive and often inaccessible to many enterprises and practitioners. Existing CPU-based inference frameworks do enable model execution, but they rely heavily on techniques such as quantisation, distillation, and pruning. These approaches trade off output quality and performance, making them unsuitable for production-grade LLM workloads.

Kompact AI was built to address this gap. Ziroh Labs designed it as a CPU-native runtime capable of running full-fidelity large language models—up to 30 billion parameters—on CPUs, without compromising accuracy or reliability. Our objective is to enable production-ready AI on widely available infrastructure and, in doing so, make AI accessible to everyone.

StartupTalky: How has 2025 shaped the growth and adoption of Ziroh Labs and Kompact AI? Could you share 2-3 key milestones from this year?

Hrishikesh Dewan: 2025 has been an important year for Kompact AI's growth. An early and foundational step in our journey was our collaboration with IIT Madras, where Kompact AI was launched on campus, and its benchmarks were initially validated.

Over the year, the platform has expanded to support 300+ large language models across text, speech, vision, and multimodal workloads—including both foundational and fine-tuned models. Alongside this, we invested heavily in platform engineering, guided by scientific advances that enable these models to run efficiently on CPUs. Many are already wired up and will be made available shortly.

 We also introduced Kompact AI One, a semantic caching–based memory module that improves LLM throughput for contextually similar queries. This goes beyond traditional prefix caching or syntactic matching approaches.

In parallel, adoption has progressed through collaborations across multiple industry verticals. Most recently, we signed a partnership with Camb AI to enable their proprietary text-to-text LLM to run efficiently on commodity and server-grade CPUs.

StartupTalky: What are the unique advantages or USPs of Kompact AI for enterprises, especially in cost-sensitive and data-regulated industries?

Hrishikesh Dewan: Accelerate Enterprise AI Adoption: Leverage Kompact AI to deploy large language models and advanced AI workloads across CPU infrastructure, enabling Enterprises to deliver faster AI solutions to clients.

• Cost-Optimised AI at Scale: Reduce infrastructure and operational costs by up to 70–80%, enabling Enterprises to offer high-performance AI solutions.
• Differentiated AI Offerings: Gain first-mover advantage in sub-50B parameter model inference on CPUs. 
• Innovation & Co-Creation: Collaborate on fine-tuning, decentralised AI models, and AI OS layer development, creating joint IP and future-ready AI solutions that expand Enterprises’ service portfolio.

StartupTalky: How does Kompact AI support developers globally, and what has the response been from the AI community so far?

Hrishikesh Dewan: Kompact AI supports developers by removing one of the biggest barriers to building AI systems today—the dependence on expensive and scarce GPU infrastructure. By enabling high-performance AI inference on CPUs, developers can build, test, and deploy AI using infrastructure they already have, across cloud and on-premise environments.

To further reduce friction, Kompact AI offers OpenAI-compatible SDKs and APIs, allowing developers to migrate existing workloads with minimal code changes. This familiarity lowers the learning curve, encourages faster experimentation, and makes it easier for teams to iterate without re-architecting their applications.

The response from the AI community has been encouraging, particularly from developers and enterprises who value ease of adoption, cost predictability, and data control. Many see Kompact AI as a practical way to move faster, experiment more freely, and take AI to production without compromising on performance or privacy.

StartupTalky: Can you share an example of how your platform has helped a client or industry deploy AI more efficiently compared to traditional approaches?

Hrishikesh Dewan: In regulated industries such as BFSI and healthcare etc, teams often face challenges in adopting AI due to GPU constraints and strict data residency requirements. Several enterprises are currently running POCs on Kompact AI, exploring use cases such as document intelligence and internal knowledge assistants within their existing environments.

Through these trials, teams are evaluating how Kompact AI enables production-grade LLM workloads to run on CPU servers, significantly reducing infrastructure complexity and enabling faster experimentation. Compared to traditional GPU-based approaches, these PoCs are helping enterprises assess more predictable cost structures, simpler operations, and stronger data control, with all data remaining within their own secure environments.

StartupTalky: What are your key plans for 2026 and beyond for Ziroh Labs, in terms of product innovation, global expansion, or partnerships?

Hrishikesh Dewan: For 2026 and beyond, Ziroh Labs focuses on deepening the platform and scaling adoption responsibly. On the product side, we are continuing to mature Kompact AI by expanding support for a broader range of models, architectures, and enterprise-grade capabilities, shaped by ongoing advances in running AI efficiently on CPUs.

On the ecosystem front, we are already working closely with OEMs, global system integrators, and data centre partners to provide Kompact AI that these organisations can take to their enterprise customers. This approach allows enterprises to adopt AI through partners they already trust, while benefiting from more predictable infrastructure, stronger data control, and greater operational maturity.

In parallel, we plan to release developer SDKs that make it easier for external developers and solution teams to run and build models on CPUs, enabling broader experimentation and accelerating enterprise-ready AI deployments at scale.

StartupTalky: As a founder, what advice would you give to startups looking to disrupt infrastructure-heavy industries like AI?

Hrishikesh Dewan: In deep technology, competition is global from day one, so startups must be prepared to compete at a global level. That means building with rigour, staying close to fundamentals, and holding the work to international standards from the start. Sincere effort, long-term thinking, and sustained hard work are essential to earning credibility and building something that can stand alongside the best in the world.

Explore more Recap'25 interviews here.