Testing a RAG Chatbot: Architecture, Strategy, and Implementation

To solve this, we implemented a three-tier testing framework for Szkolenia Skin Atelier. It separates fast offline unit checks, live AI evaluation assertions, and browser-level End-to-End (E2E) automation.

• →Vitest for fast, isolated unit and integration tests with TypeScript support.
• →Playwright for cross-browser E2E automation and visual regression testing.
• →Google Gemini API for live AI response quality evaluation and semantic assertions.
• →Custom test runners to isolate Vitest from Playwright and prevent namespace collisions.

The Three-Tier Testing Architecture

1. Offline Unit & Integration Tests (Vitest)

Offline tests run in less than 500 milliseconds and cost $0.00. They verify core application logic, custom rate limiters, and mock routing handlers.

• →Rate Limit Testing (test/unit/rate-limit.test.ts): uses fake timers to prove the sliding-window IP limiter allows exactly 10 requests per minute, blocks the 11th with HTTP 429, and restores access after the window expires.
• →Vector Math Validation (test/unit/rag-retrieval.test.ts): asserts identical vectors score 1.0, orthogonal vectors score 0.0, and opposites score -1.0.
• →Mock API Integration (test/unit/gemini-mock.test.ts): mocks the ai package and Nodemailer, verifying the chat API retrieves context, builds prompts, and attempts lead email delivery without real network calls.

2. Live AI Quality Evaluation Tests (Vitest)

These tests interact with the live Google Gemini API. Because LLM outputs vary, we use semantic assertion techniques instead of exact string matching.

• →Compliance Verification (test/live/gemini-live.test.ts): streams Gemini output, parses SSE JSON, and asserts the response is written in Polish and includes the required training program links.
• →Semantic Similarity Verification (test/live/gemini-live.test.ts): compares response embeddings against a golden standard and passes only if the cosine similarity exceeds 0.75.

3. Playwright E2E Browser Tests (Playwright)

E2E tests spin up the Next.js app on port 3001 and verify visual layouts, responsive viewport behavior, and parent-child iframe communication.

• →Theme Color Verification (test/e2e/chatbot-ui.spec.ts): reads DOM CSS properties to verify branding colors and chat bubble styles.
• →Iframe Resizing & Messaging (test/e2e/chatbot-ui.spec.ts): ensures the chat toggle expands the iframe from 120x120px to the correct desktop/mobile viewport dimensions.
• →Conversational User Flows (test/e2e/chatbot-ui.spec.ts): simulates a query, verifies user bubbles appear, and checks the loading bounce indicator.

Runner Configuration & Isolation

To prevent test runner conflicts, we isolated Vitest from Playwright. Playwright uses a custom global test namespace, so loading it inside Vitest can crash the suite.

• →Default config targets test/unit only and excludes test/e2e.
• →Live test separation uses vitest.live.config.ts so paid AI checks do not run during local unit watches.