Textual response generation for multimodal task-oriented dialog systems, which aims to generate the appropriate textual response for the given multimodal context, is a pivotal yet challenging task. Although existing studies have shown fruitful progress, they still suffer from two critical limitations: 1) ignore the relation knowledge, and 2) lack the representation-level regularization. To address these limitations, we propose the novel multimodal dialog system (HRMD) with dual semantic knowledge composition and representation-level semantic regularization.
Specifically, HRMD first acquires the related attribute knowledge and relation knowledge for the given multimodal context from the knowledge base, where the non-intuitive relation knowledge is extracted by the n-hop graph walk.  Subsequently, HRMD composes the multimodal context and acquired dual semantic knowledge to obtain the latent composed response representation, where the attribute and relation knowledge are composed at the input token level and the intermediate representation level, respectively. Moreover, apart from the common output-level supervision  for the response generation, HRMD also incorporates the representation-level semantic regularization, where a set of latent query variables are devised to absorb the semantic information from the composed response representation and the ground truth response representation, respectively, and hence fulfill the representation-level regularization. Notably, this regularization allows HRMD to further utilize the composed response semantic representation to enhance the response generation. Extensive experiments on a public dataset verify the superiority of our proposed HRMD. Meanwhile, we release the  codes and parameters to facilitate the research community.