March 20, 2023

NEC won the 3rd “AIO: Japanese Quiz AI Competition”, held from August to December 2022. This second consecutive victory follows the victory in the previous competition, which was the first time NEC participated in this event. What is this competition all about? What led to this feat? We spoke with the researchers about the details.

Correctly answering 938 questions out of 1,000

― What kind of competition is the “AIO: Japanese Quiz AI Competition”?

Oyamada: It’s a competition voluntarily held by a group of researchers from Tohoku University, RIKEN, NTT, and other organizations with the aim to promote studies into question answering technologies. Participating AIs compete on how correctly they can answer, within six hours, without the Internet connection, the 1,000 questions from common quizzes given to humans. The usable computational functions are regulated, and the AIs are to execute their best performance within such restrictions. This event has been held three times up to this date, and NEC participated in the 2nd (2021) and 3rd (2022) competitions and won both times. In the 2nd and 3rd competitions, NEC’s AI answered correctly to 839 and 938 questions, respectively.


Akimoto: While we managed to win two years in a row, the 3rd competition was a narrow victory, so we couldn’t let down our guard till the end. Notably, I had this stomach-churning happening where we noticed a serious bug a day before the final submission date and had to make adjustments.


― Why did you decide to participate in this competition?

Takeoka: Mr. Akimoto and I came across information about this competition during a natural language processing conference that we participated in together, and that’s how we made up our mind to enter the competition. We happened to be studying question-answering technology at the time for the NEC Data Enrichment (https://jpn.nec.com/solution/dataenrichment/) service, which our team is working on, so entry into such a contest seemed to be a good opportunity both for efficient learning and to objectively assess our AI’s performance.


Oyamada: The Data Science Research Laboratories have a system that allocates budgets to new, productive research devised by researchers on top of their regular research activities. We applied for that system and have been working on the research as part of our corporate activities.

Developing precision AI models by decomposing functions

― What do you think determined the victory?

Akimoto: In our first time participating in this competition, we started out as total newbies in question-answering technology, so Mr. Takeoka and I tried to catch up by reading up as many research papers as possible. It was a giant step that we looked thoroughly into the best possible methods as of that point of time and managed to properly implement them.


Oyamada: Our results in the 2nd competition were pretty good, so for the 3rd competition, the architecture that we adopted in that competition was set as the baseline. This made the 3rd Competition a fierce battle. That is because all other competitors start from the same line as us. In our team, Mr. Takeoka and Mr. Akimoto used artifice to boost the performance.


Takeoka: Our AI models can be broadly divided into three components: the first one retrieves information relevant to the question, the second one narrows down the retrieved information, and the last one generates answer candidates from the information and decides the answer by verifying them. Initially, we took too much time in the second filtering part, which we struggled to improve. The third part, handled by Mr. Akimoto, uses a unique structure that derives the answer by aptly utilizing multiple models. Additionally, the inclusion of a mechanism that checks whether the final answer is correct at the end is a technically interesting aspect. It was only our team that had a feature that would do a final check.


Akimoto: Before including the final check function, our AI frequently made trivial mistakes such as an error in orthography. So, we combined two modules, one that outputs multiple answer candidates and the other that subsequently checks the answer candidates and singles out the best one.


Oyamada: While you can do something similar by simply applying a commonly-available, large-scale AI model, decomposing the task into multiple parts to be taken care of by separate models instead of having a single model do everything makes it easier for each model to exert its functionality. This trend has recently manifested itself in the language model research community, but this time we were able to see it first-hand.

To an innovation that efficiently reads and comprehends a large volume of complex statements

― Were there insights from working on the competition entry that you can take away for your regular operations?

Oyamada: We got to reaffirm the importance of data. For the 3rd Competition, we were initially repeating model modifications and not seeing as much improvement in accuracy as we had hoped for, but what we realized through close examination is that gathering better-quality data contributes more to the actual accuracy. This was both a very effective insight for business and a fruitful discovery from a manager’s perspective.


Akimoto: Yes. Data is very important; we acknowledged that there is a phenomenon that if you give unnecessary information, the AI model will not be able to answer questions that it previously answered correctly. Realization of that negative effect was also one of the takeaways.


Takeoka: That is exactly what has been mentioned in recent research papers, and possibly a theme that is to be a hot topic worldwide. We were also able to witness cases where the AI model remembers, in its learning process, time-variable information in a fixed way, such as “prime minister of Japan,” and ends up making a mistake.


― In what ways do you hope to apply the question answering technology?

Oyamada: First, we want to apply what we gained and developed through our efforts for the competition to our team’s current commercialization project, the NEC Data Enrichment. This service automatically adds attribute information, such as number of employees and prices, to product and company information by searching the Web. The know-how that we gained through the competition can further evolve this solution in terms of efficiency and accuracy.

Another interesting application may be a solution that supports the interpretation and understanding of massive amounts of complicated literature, for example, legal documents.


Takeoka: I would also like to develop an AI model that can support the understanding of writing that is not up my alley, like legal and administrative documents. The image is like having a dialogue with the writing. For example, if the question-answering technology can get between the writing and the reader, and translate, that may enable even complete non-experts to read difficult literature. We would like to develop such solutions.


Akimoto: I am considering taking a step further, to apply the question-answering technology to what I wrote as well. The thing is, I don’t have good memory and forget things easily. So, I leave memos, but then I forget about writing them. When this happens, it is a great loss for the future me―I need to rethink what I have already thought over. So, I am thinking about having the AI read what I wrote previously, like journals and memos, and create a system where I can seamlessly and smoothly take out information not only from what I have memorized, but also from previous writings with the help of AI. I expect the question-answering technology to be a gateway for such applications.


― I see. Congratulations on the two consecutive wins. Would you enter the competition again if there is one?

Oyamada: I think so. We would like to reach out to various subjects in addition to quizzes.