GitHub projects

Creating semantic cache to store responses from LLM queries.

Cloud-Native LLM Routing Engine. Improve LLM app resilience and speed.

Cost-effective LLM development in any cloud (AWS, GCP, Azure, Lambda, etc).

Stream large multimodal datasets to achieve near 100% GPU utilization. Query, visualize, & version control data. Access data w/o the need to recompute the embeddings for the model finetuning.

Self-hosted context engine for AI agents with persistent conversation memory and recall. Works as a drop-in OpenAI-compatible proxy, OpenClaw plugin, or memory SDK — no code changes required.

Web framework to create vertical AI agents. FastAPI based, plugin system inspired to WordPress, admin panel, vector DB included

Deploy a ML inference service on a budget in less than 10 lines of code.

A Reliable Open Source AI studio to build core infrastructure stack for your LLM Applications. It allows you to gain visibility, make your application reliable, and prepare it for production with features such as caching, rate limiting, exponential retry, model fallback, and more.

Type-Safe Markdown-based Agents

Visualizes RAG knowledge gaps and "blind spots" using 2D UMAP clustering and density detection.

Production-grade SDK for observability, automated evaluations and prompt management with sub-100ms guardrails for LLM/agent workflows.

Open-source AI tracing framework built on OpenTelemetry for deep observability across agentic and LLM workflows.

CLI tool for debugging and benchmarking RAG retrieval. EXPLAIN ANALYZE for your retrieval layer.

Lightweight Go CLI that tracks AI API quota usage across 7 providers (Anthropic, OpenAI, GitHub Copilot, MiniMax, and more). Background daemon, <50MB RAM, zero telemetry, SQLite storage.

The open standard for data logging

OpenTelemetry-based observability and monitoring for LLM and agents workflows.

Open source LLM observability platform. One line of code to monitor, evaluate, and experiment with features like prompt management, agent tracing, and evaluations.

Always know what to expect from your data.

Deterministic verification protocol for LLM outputs using 8 formal verification engines (SymPy, Z3, AST, SQLGlot). Prevents hallucinations through mathematical proofs rather than statistical methods.

Evaluate, monitor, analyze, and improve machine learning and generative models from pre-production to production. Ship more ML and LLMs into production, and monitor ML and LLM metrics like hallucination, PII, and toxicity.

Regression testing for AI agents. Snapshot behavior, detect tool-call and output regressions, with golden-baseline diffing and LLM-as-judge scoring. Supports LangGraph, CrewAI, OpenAI, Claude, and any HTTP API.

"Batteries included" logging solution for your Azure OpenAI instance.

A Python Machine Learning Pentesting Toolbox for Adversarial Attacks. Works with LLMs, DNNs, and other machine learning algorithms.

Open-source confidential AI framework for secure LLM deployment with data privacy, providing hardware-enforced isolation using Intel TDX and NVIDIA Confidential Computing.

CLI tool for running coding agents inside hardware-isolated microVMs with snapshot isolation, egress control, and MCP authorization.

A Kubernetes operator that simplifies serving and tuning large AI models (e.g. Falcon or phi-3) using container images and GPU auto-provisioning. Includes an OpenAI-compatible server for inference and preset configurations for popular runtimes such as vLLM and transformers.

Deploy and scale machine learning models on Kubernetes. Built for LLMs, embeddings, and speech-to-text.

Replace OpenAI GPT with another LLM in your app by changing a single line of code. Xinference gives you the freedom to use any LLM you need. With Xinference, you're empowered to run inference with any open-source language models, speech recognition models, and multimodal models, whether in the clou

LLMs on Ray - RayLLM (Archived)

FastAPI framework to build production-grade LLM applications

Serverless LLM apps on Production with Jina AI Cloud (Archived)

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

Serve, optimize and scale PyTorch models in production (Archived)

A flexible, high-performance serving system for machine learning models.

A machine learning model serving framework with dynamic batching and pipelined stages, provides an easy-to-use Python interface.

Build multimodal AI services via cloud native technologies · Model Serving · Generative AI · Neural Search · Cloud Native

Real-time inference for Stable Diffusion - 0.88s latency. Covers AITemplate, nvFuser, TensorRT, FlashAttention. (Archived)

Port of OpenAI's Whisper model in C/C++

is a 4x faster and low-memory usage drop-in cli replacement that supports word-level timestamps and VAD filter

Large Language Model Text Generation Inference

OpenAI-compatible LLM inference server for Apple Silicon using MLX. 2-4x faster than Ollama with tool calling and prompt caching.

Open-source iOS/Android app running LLMs on-device via llama.cpp. Voice (Whisper), vision, image gen, tool calling — fully offline.

OpenAI compatible API for LLMs and embeddings (LLaMA, Vicuna, ChatGLM and many others)

Kubernetes operator for LLM inference with pluggable runtimes (llama.cpp, PersonaPlex/Moshi, generic), multi-GPU sharding, NVIDIA CUDA and Apple Silicon Metal support, and GGUF/MLX/SafeTensors model formats.

Running large language models on a single GPU for throughput-oriented scenarios. (Archived)

fast inference engine for whisper in C++ using CTranslate2.

serving the OpenAI CLIP model

fast inference engine for Transformer models in C++