- Foundation Models
- GPT-4o / GPT-4o-mini (OpenAI), Claude 3.5 Sonnet / Haiku (Anthropic), Gemini 1.5 Pro / Flash (Google), Llama 3.1 (Meta, self-hosted), Mistral Large / 7B (self-hosted)
- LLM Orchestration
- LangChain (complex pipelines, tool use), LangGraph (stateful agent workflows), LlamaIndex (RAG-focused), direct SDK for simple integrations
- Embedding Models
- text-embedding-3-large (OpenAI), Cohere Embed v3, BGE-M3 (open-source, self-hosted), Voyage AI
- Vector Databases
- Pinecone (managed, production-scale), Weaviate (self-hosted, metadata filtering), Chroma (dev/small-scale), pgvector (PostgreSQL extension)
- Reranking
- Cohere Rerank, cross-encoder models (SBERT), LLM-based reranking for high-accuracy requirements
- Agent Frameworks
- LangGraph (stateful, production agents), AutoGen (multi-agent), CrewAI (role-based agents), custom ReAct implementation
- Document Parsing
- Unstructured.io (multi-format: PDF, Word, HTML, tables), LlamaParse (complex PDFs), Apache Tika, custom parsers
- Evaluation Framework
- RAGAS (RAG evaluation: faithfulness, relevance, context precision), LangSmith (tracing + evaluation), custom eval harnesses
- Streaming & API
- Server-Sent Events (SSE), FastAPI (Python backend), React with streaming state management, WebSockets for agents
- Monitoring & Observability
- LangSmith / Langfuse (LLM tracing), Helicone (cost monitoring), custom Grafana dashboards, Sentry, Datadog
- Data Privacy & Security
- PII detection and redaction (Presidio), prompt injection detection, output safety classification (Llama Guard), data residency controls, private deployment on AWS/Azure/GCP
- Infrastructure
- AWS (ECS Fargate, Lambda, S3, OpenSearch), Azure (OpenAI Service), GCP (Vertex AI)