来自于Attention is all your need！

PE(pos,2i) = sin(pos/10000**(2i/dmodel))

PE(pos,2i+1) = cos(pos/10000**(2i/dmodel))


position_enc = np.array([
        [pos / np.power(10000, 2 * (j // 2) / emb_dim) for j in range(emb_dim)]
        if pos != 0 else np.zeros(emb_dim) for pos in range(n_position)])


position_enc[1:, 0::2] = np.sin(position_enc[1:, 0::2]) # dim 2i
position_enc[1:, 1::2] = np.cos(position_enc[1:, 1::2]) # dim 2i+1

参考

  • https://github.com/CyberZHG/keras-pos-embd/blob/master/keras_pos_embd/pos_embd.py