With the recent advances of the sequence-to-sequence framework, generation approaches for the short text conversation (STC) become attractive. The traditional sequence-to-sequence approaches for the STC often suffer from poor diversity and general reply without substantiality. It is also hard to control the topic or semantics of the selected reply from multiple generated candidates. In this paper, a novel external-memory-driven sequence-to-sequence learning approach is proposed to address these problems. A tensor of the external memory is constructed to represent interpretable topics or semantics. During generation, a controllable memory trigger is extracted given the input sequence, and a reply is then generated using the memory trigger as well as the sequence-to-sequence model. Experiments show that the proposed approach can generate much richer diversity than the traditional sequence-to-sequence training with attention. Meanwhile, it achieves better quality score in human evaluation. It is also observed that by manually manipulating the memory trigger, it is possible to interpretably guide the topics or semantics of the reply.