Leveraging multilingual neural language models for on-device natural language understanding

Learning cross-lingual word representations is an effective approach for developing multilingual models. In this work, we lay the groundwork and present preliminary results on learning cross-lingual representations appropriate for deployment to edge devices. Specifically, we learn cross-lingual representations using multilingual language models and use these to seed different parts of a Neural Natural Language Understanding (NLU) model that is designed for an on-device deployment. This is followed by fine-tuning on supervised datasets to perform NLU. We present systematic experiments that show up to 10% relative improvement in NLU error rates across different dataset sizes. We perform experiments for three languages and confirm the effectiveness of using cross-lingual representations for both multi- and monolingual NLU models for low-resource languages that have scarce task-specific annotated data. The effectiveness of cross-lingual transfer indicated by these preliminary experiments will be used in our future work for creating multilingual on-device models.