NLP-driven text summarization of Japanese documents
Tech stack: CRF, LSTM, BERT, T5, GPT2
Development team
Team size: 2 people
Background
It is common for professionals to process long documents containing hundreds of pages daily. The huge workload can overwhelm staff and significantly decrease their efficiency. However, dealing with the human language has always been a considerable challenge for AI engines due to the former’s complex and nuanced nature.
Our client, a Japan-based leading global provider of integrated solutions in printing, communications, security, packaging, décor materials, and electronics, approached GEM with a question: how can we derive meaningful information and insights from those documents with both speed and accuracy?
This request prompted an interesting challenge for GEM’s AI engineers and urged them to leverage both their tech expertise and problem-solving skill to create a powerful and satisfactory AI engine.
Challenges
To ensure the perfect functionality of the engine, we addressed several challenges relating to the technical and linguistic aspects.
The first challenge was cross-domain text processing. Traditional machine learning models would require separate development efforts for each domain, which may lead to labor-intensive processes.
The second challenge is the complex nature of the Japanese language. Japanese presents a unique set of challenges due to its sophisticated structure and writing systems (including hiragana, katakana, kanji). Analyzing Japanese text involves identifying and parsing multiple scripts and understanding different parts of speech. Moreover, while general Japanese text summarization AI exists, creating domain-specific summarization models adds another layer of complexity to the project.
Solution
To address both challenges, GEM’s team implemented the transfer learning technique to transfer knowledge from one domain to another efficiently. We opted for some of the latest language models including T5 (Text-to-Text Transfer Transformer) by Google, BERT, and GPT2 in this approach.
First, we needed to extract key phrases from the source document. Then, we labeled those key phrases and trained a binary machine learning classifier to make the text summarization. Finally, in the testing phases, we carried out classification for the created keywords and sentences.
In addition, we performed feedback-based improvements after we receive the client’s reviews of the engine’s usability every week.
Output
After months of development, GEM’s team developed an NLP model capable of performing accurate and efficient text summarization. The first domain that it was implemented in was banking and the engine yielded an impressive accuracy rate.
Impacts
When being applied in Japanese businesses and organizations, the text summarization engine yielded the following positive impacts:
- Time efficiency: The AI solution automates document processing to cut down on manual efforts and allow employees to focus on value-added tasks.
- Enhanced decision-making: It grants quick access to key insights from documents, thus facilitating faster, informed decision-making and driving organizational agility
- Improved accessibility and transparency: Concise summaries help staff across different departments access knowledge and information quickly, thus eliminating confusion and data silos.
Closing remarks
This project is a testament to the growing capability of technology in handling sophisticated challenges in our world. By harnessing cutting-edge advancements such as NLP, GEM can empower professionals to achieve streamlined information processing and decision-making. As a result, we foster an innovative business environment where technology augments human capabilities and unlocks new frontiers of efficiency and productivity.
