The cardiovascular disease (CVD) is one of the major causes of mortality worldwide. Auscultation of heart sounds or phonocardiograms (PCGs) analysis, which is an efficient and non-invasive way, has been shown to be promising and played an important role in preliminary CVD diagnosis. In this study, a deep learning-based PCG classification method is proposed, which is mainly comprised three steps: pre-processing, PCG patches classification using a novel 1-D deep convolutional neural network (CNN), and final predicting of PCG recordings based on the patch-level results. In order to maximize the information flow within the CNN, a block-stacked style architecture with clique blocks is employed, and in each clique block a bidirectional connection structure is utilized. Using the stacked blocks, the proposed CNN achieves both spatial and channel attention, which leads a superior classification performance. Besides, a novel separable convolution with inverted bottleneck is introduced to efficiently decouple features' dependency between spatial and channel-wise dependency of features. Experiments on PhysioNet/CinC 2016 reveal a superior classification performance and the advantage in parameter efficiency of the proposed method comparing to state-of-the-art methods.