When subjected to external stimuli such as mechanical loading, atoms in disordered solids respond heterogeneously. Due to lack of representations to resolve the subtle packing difference around atom sites and approaches to deal with the long-range correlation involved, it is hard to quantitatively predict this heterogeneous, site-specific response solely from the structure. Here, by designing a robust hierarchical machine learning framework, we show that it is possible to predict the mechanical heterogeneity in disordered solids a priori, directly from the quenched structural state itself. We encyclopedically create a large pool of 810+ site descriptors, from 40+ sets of structural measures, spanning topological and chemical short- and medium-range order, and develop a novel hierarchical scheme to further extend the studied scale to an unprecedentedly long-range while still retaining good interpretability and generality. Impressive predictability is achieved in a fairly large strain regime, suggesting a long-lived inheritance of the quenched state until later obstructed by shear banding. The framework is robust over a range of compositions and processing conditions and can well detect the site environments tuned by these conditions. We also identify a bag of promising structural signatures unrevealed previously, with their predictability exhaustively benchmarked and discussed. This hierarchical learning framework is general and could potentially be applied to decode structural origin of any site-specific properties in the family of disordered materials.