Background
AI chatbots have become the trusted tool for expediting various business functions, from customer services to supply chain management and supply chain.
However, crafting question-answer pairs for these chatbots can be labor-intensive, especially when dealing with extensive instruction manuals in complex industries like banking or logistics.
To address this challenge, GEM proposed training an AI model to analyze Japanese instruction manuals efficiently. The model would scan documents to generate questions and answers. By automating this phase in the development process, our solution enabled adopters of AI chatbots to quickly develop chatbots that catered to the needs of any industry or any business function.
Challenges
The challenges faced in this project included:
- Transfer learning complexity: Transfer learning in natural language processing (NLP) is inherently more complex than in visual learning due to the nuanced nature of language comprehension and generation.
- Lack of precedent: Limited previous application of advanced language models (T5 – Text-to-Text Transformer by Google, BERT, and GPT2) for question generation posed a challenge. Therefore, our team needed to spend more time researching and finding solutions.
- Sophistication of the Japanese language: The intricacies of the Japanese language, such as a complex writing system, cultural sensitivities, and various grammatical rules, presented an additional layer of challenge. This issuer necessitated thorough research and adaptation strategies to effectively analyze and generate questions from Japanese text.
Solution
To tackle the complexities of transfer learning in NLP, the team conducted in-depth research and extensive experimentation and refinement to optimize model performance for question generation. We also had thorough linguistic analyses to understand the nuances of Japanese text, so that we could develop specialized preprocessing techniques tailored to the language’s unique characteristics.
The team also invested additional time and effort into researching and experimenting with various approaches, with a goal to effectively utilizing T5, BERT, and GPT2 for the project.
Tech stack
- Cutting-edge language models:
- T5 (Text-to-Text Transfer Transformer) by Google
- BERT
- GPT-2
- Transfer learning capabilities utilized for knowledge transfer between contexts
- Supporting tools and frameworks for data preprocessing, model training, and evaluation in natural language processing tasks
Output
The output was a powerful AI model that fulfills its role. First, it processed the whole paragraph to identify topically important sentences, based on which questions can be generated, by leveraging POS Tagger and feature extraction. The AI then identified question words with the selected questions and rearrange the sentence to generate questions
When being deployed to process banking instruction manuals, the AI had highly accurate output.
Impacts
With the question-answer generator, the client enjoyed the following positive changes when developing an AI chatbot:
- Accelerating the chatbot development timeline
- Minimizing errors and improves the overall quality of chatbot interactions
Closing remarks
The project highlighted GEM’s team’s efforts in addressing the challenges posed by the complexity and nuance of the human language. By leveraging multiple tech tools and conducting thorough research, we delivered a satisfactory solution to our client and proved our commitment to meeting the evolving needs of our clients.
