Domain-specific LLM adaptation: Bridging personalization and efficiency through synthetic data and optimization

Large Language Models (LLMs) have demonstrated exceptional capabilities but face two critical deployment challenges: high computational costs and scarcity of personalized domain training data. We address these dual challenges through a comprehensive framework that combines synthetic data generation with inference optimization techniques. Our approach employs LLMs for zero-shot and few-shot synthetic dataset creation while applying structural pruning, knowledge distillation, quantization, and prompt caching for computational efficiency. We evaluate three architectural paradigms: encoder-only, encoder-decoder, and decoder-only on synthetic building permit classification and assess optimization techniques on public benchmarks. Our systematic evaluation reveals that strategic architectural selection based on task characteristics is more critical than model complexity: encoder-only models provide superior efficiency-accuracy trade-offs for high-throughput scenarios, demonstrating that understanding problem requirements enables effective deployment without requiring the largest available models. Knowledge distillation emerges as the key optimization technique for personalized domains, recovering pruning-induced performance degradation with Llama3-8B+KD achieving 87.1% accuracy at 20% pruning while exceeding unpruned baselines. Complementary optimization strategies, including dynamic caching (5-18% latency reduction with zero performance loss) and hardware acceleration (up to 200× speedup) enable flexible deployment configurations for domain specific applications.