Click here to flash read.
Traditionally, approximate dynamic programming is employed in dialogue
generation with greedy policy improvement through action sampling, as the
natural language action space is vast. However, this practice is inefficient
for reinforcement learning (RL) due to the sparsity of eligible responses with
high action values, which leads to weak improvement sustained by random
sampling. This paper presents theoretical analysis and experiments that reveal
the performance of the dialogue policy is positively correlated with the
sampling size. To overcome this limitation, we introduce a novel
dual-granularity Q-function that explores the most promising response category
to intervene in the sampling process. Our approach extracts actions based on a
grained hierarchy, thereby achieving the optimum with fewer policy iterations.
Additionally, we use offline RL and learn from multiple reward functions
designed to capture emotional nuances in human interactions. Empirical studies
demonstrate that our algorithm outperforms baselines across automatic metrics
and human evaluations. Further testing reveals that our algorithm exhibits both
explainability and controllability and generates responses with higher expected
rewards.
