The degradation behavior of aggregate skeleton in stone matrix asphalt mixture was investigated based on theoretical analysis, laboratory test and field materials evaluation. A stress-transfer model was established to provide the fundamental understanding of the stress distribution and degradation mechanism of stone matrix asphalt (SMA) aggregate skeleton. Based on the theoretical analysis, crushing test and superpave gyratory compactor (SGC) test were used to evaluate the degradation behavior of aggregate skeleton of SMA. To verify the laboratory test results, gradation analysis was also conducted for the field materials extracted from SMA pavements after long-time service. The results indicate that the degradation of SMA aggregate skeleton is not random but has fixed internal trend and mechanism. Special rule is found for the graded fine aggregates generated from coarse aggregate breakdown and the variation of 4.75 mm aggregate is found to play a key role in the graded aggregates to form well-balanced skeleton to bear external loading. The variation of 4.75 mm aggregate together with the breakdown ratio of aggregate gradation can be used to characterize the degradation behavior of aggregate skeleton. The crushing test and SGC test are proved to be promising in estimating the degradation behavior of SMA skeleton.