How it works
Second Opinion
This demo showcases "Edge AI"—running a Large Language Model entirely locally inside your web browser. Using WebGPU hardware acceleration, your device processes the reviews and generates the summary without ever sending data to a server.
Business applications
Because inference happens completely on-device, highly sensitive data (like internal HR reviews, personal medical notes, or proprietary code) never leaves the user's local network. This eliminates cloud-based security risks.
2
Zero Cloud Computing Costs
By offloading the computational workload to the user's hardware (laptop or smartphone GPU), businesses can scale AI features to millions of users without incurring exponential AWS/cloud API costs for LLM inference.
High-level technical workflow
1
WebGPU Engine Initialization
When initiated, the application checks for WebGPU support. It then utilizes Apache TVM and WebLLM to compile the AI model's shaders specifically for the user's local graphics hardware.
2
Weights Download & Caching
The model weights are downloaded from a CDN and permanently cached in the browser's IndexedDB. Subsequent visits load the model instantly from local memory.
3
Local Streaming Inference
The DOM text is scraped, formatted into a system/user prompt array, and processed locally. The generated tokens are streamed back to the UI in real-time as they are computed by the local GPU.