本研究では，マルチモーダル言語理解，マルチモーダル言語生成，Sim2Real転移学習と介助犬タスクでの評価を行った．理解班では，Vision-and-Language Navigationタスクにおいて，敵対的摂動更新アルゴリズム Momentum-based Adversarial Trainingを構築した．生成班では，動画から将来の状況を説明する文を生成するfuture captioning手法を構築し，既存手法を上回る結果を得た．Sim2Real班では，生活支援ロボット評価フレームワークを構築し，指示文生成とタスク実行を自動化した．
In this study, our objectives are (a) robust multimodal language understanding through adversarial data augmentation, (b) multimodal language generation, and (c) evaluation in the assistance dog tasks.
We first focused on the Vision-and-Language Navigation task and developed the Momentum-based Adversarial Training (MAT) algorithm. We applied MAT to the standard benchmark test, ALFRED, and obtained successful results. We also worked on the task of generating descriptions about future situations. The main novelty of our proposed method lies in the use of Relational Self-Attention as the attention mechanism. Experimental results show that our method outperformed existing methods in standard metrics. We applied the multimodal language understanding and generation methods into a simulator, enabling on-the-fly instruction generation. As a result, we established a robot evaluation framework that does not require manual intervention in task generation, execution, and evaluation.

研究種目 : 基盤研究 (B) (一般)
研究期間 : 2020～2022
課題番号 : 20H04269
研究分野 : 機械知能, 知能ロボティクス, マルチモーダル言語処理