Zero-Shot Recognition Accuracy Reaches 95.4%: MIT and Others Propose CLaC Model for Cross-Modal Alignment Between Material Structures and Language

A critical task in materials discovery is screening candidate materials with target properties from a vast chemical and structural space. In recent years, graph neural networks have made significant progress in tasks such as materials property prediction and structure generation by learning the three-dimensional atomic structures of crystals. However, these models typically rely on predefined property labels or numerical conditions, making it difficult to directly understand natural language descriptions of material requirements; while large language models excel at processing text, they lack accurate representations of crystal spatial structures and their physical properties.

Combining crystal structures with natural language offers new avenues for materials discovery, but the limited scale of existing crystal databases and the concentration of academic literature on a few popular materials result in insufficient correspondence between text descriptions and material properties, constraining the training effectiveness of multimodal models.

To address this issue, researchers from institutions including the Massachusetts Institute of Technology (MIT), the Ulsan National Institute of Science and Technology (UNIST) in South Korea, the University of Illinois Urbana-Champaign, and the MIT-IBM Watson AI Lab proposed the Contrastive Language-Crystals (CLaC) model. This model utilizes large language models to generate synthetic narratives based on material properties, mapping crystal structures and natural language into a shared representation space through contrastive learning. Experiments demonstrate that, compared to models trained directly on academic literature text, CLaC exhibits significant advantages in zero-shot materials understanding and natural language-guided materials retrieval tasks.

The relevant research results, titled "Scaling multimodal materials representation learning with synthetic narratives," have been published in Nature Communications.

Paper link:

https://www.nature.com/articles/s41467-026-76378-7#Sec10

126,000 Crystal-Text Paired Data Samples, Expanding Materials Semantic Coverage

Training multimodal crystal models first requires establishing a correspondence between crystal structures and natural language. Although existing materials databases have accumulated large amounts of crystal structures and computed properties, relevant texts are mainly scattered in academic papers and concentrated on a few well-studied materials, making it difficult to cover diverse crystal structures or ensure that texts contain complete material property information.

To this end, the study collected approximately 126,000 crystal material data entries based on the Materials Project database, and constructed crystal graph structures and natural language descriptions respectively. Crystal structures were converted into graph data using the Jarvis-tools, representing the three-dimensional structure via atoms and their interactions; for the text component, the researchers utilized a previously developed GPT synthetic narrative dataset, using large language models to generate descriptive text based on computed properties from the materials database.

Unlike simply converting chemical formulas or elemental compositions into text, synthetic narratives are based on computed material properties, covering structural features, physical properties, functional behaviors, and potential applications. For example, based on known data such as the composition, band gap, and density of a certain crystal, descriptions containing corresponding physical characteristics and application information are generated, establishing a correspondence between crystal structures and richer materials semantics.

To verify the effectiveness of synthetic narratives, researchers also constructed a control dataset from academic literature, extracting relevant paragraphs from the full text of approximately 4 million scientific papers using chemical formulas as search criteria, ultimately obtaining approximately 400,000 material-natural language sentence paired data samples, but covering only about 400 inorganic compounds. In comparison, the number of distinct crystal types covered by GPT synthetic narratives is approximately 300 times that of the academic literature data.

The two types of text data were used to train CLaC-Synthetic (synthetic narrative version) and CLaC-Paper (academic literature version) respectively, to compare the impact of different data sources on model performance. The datasets were divided into training, validation, and test sets in an 8:1:1 ratio.

Combining Graph Neural Networks and Language Models to Align Crystal Structures with Natural Language

CLaC draws on the contrastive learning methods of CLIP in the visual language domain, jointly learning crystal structures and text descriptions to map both types of information into a shared representation space, enabling the model to calculate semantic similarity between materials and natural language.

CLaC employs a dual-encoder architecture, consisting of a crystal graph encoder, a text encoder, and a multimodal projection module. The crystal graph encoder is responsible for extracting information such as atomic composition, local coordination environment, and three-dimensional geometric structure, while the text encoder processes natural language descriptions of material properties, functional features, and application requirements. After converting both types of information into vectors, they enter a unified latent space through the projection module, establishing a correspondence between crystal structures and text semantics.

In terms of crystal structure encoding, the study employs the PaiNN (Polarizable Atom Interaction Neural Network) graph neural network, which learns spatial relationships between atoms through equivariant message passing mechanisms, capturing local atomic environments while preserving geometric information. CGCNN and the pre-trained interatomic potential model ORB were introduced as controls.

The text encoders compared included pre-trained language models such as SciBERT, MatSciBERT, OPT-125M, and Galactica-125M, covering both bidirectional encoding and autoregressive architectures. Based on subsequent experimental results, the researchers ultimately adopted the combination of PaiNN and Galactica-125M as the primary encoder configuration.

The core of CLaC is cross-modal contrastive learning. During training, crystal graphs and text descriptions of the same material are treated as positive pairs, while combinations of structures and texts from different materials form negative pairs. By leveraging the InfoNCE contrastive loss, the model increases the vector similarity of correct pairs and decreases the similarity between mismatched samples, gradually learning the relationships among crystal structures, physical properties, and natural language.

Given the limited paired data, the study also incorporates intra-modal contrastive learning, enabling the model to recognize different structural representations of the same crystal and different textual descriptions of the same material. Crystal graphs are augmented through atomic position perturbations and 0–5% lattice strain; text versions are generated by random deletion and shuffling of sentence order, with a retention probability of 50% for each sentence. These methods enhance the model’s robustness to minor structural variations and textual expression differences. The researchers did not employ isovalent substitution, as it might alter the coordination environment, electronic structure, and actual properties of the materials.

During pretraining, the crystal graph encoder is trained from scratch, while the text encoder is initialized with pretrained weights. The model utilizes a 512-dimensional joint embedding space, the AdamW optimizer with a learning rate of 2×10⁻⁵, an effective batch size of 256, and is trained for a maximum of 5 epochs.

After pretraining, CLaC can convert natural language queries into vectors and compare them with pre-computed crystal vectors in the material database via similarity matching, thereby screening candidate materials with target properties, without requiring model retraining for each property. However, CLaC itself does not directly generate crystal structures; its primary function is to establish the connection between language and crystal structures, supporting subsequent inverse design of materials.

From Zero-Shot Material Understanding to Natural Language Retrieval: Validating Cross-Modal Learning Effectiveness

To evaluate CLaC’s performance, researchers designed multiple zero-shot experiments focusing on the model’s capabilities in crystal understanding, natural language retrieval, and physical property representation, comparing it against baselines such as untrained models, Gemma-3-1b-it, and CLaC-Paper.

First, in the zero-shot multimodal understanding experiment, the model was required to select the correct description from multiple candidate texts based on crystal structures, involving three task types: elemental composition, crystal system, and oxide type. Results showed that CLaC-Synthetic achieved an accuracy of 95.4% ± 0.2% in elemental composition classification, significantly higher than Gemma-3-1b-it’s 17.8% and CLaC-Paper’s 13.6%; the accuracy for oxide classification reached 66.4% ± 0.4%, also outperforming other models.

However, CLaC-Synthetic achieved an accuracy of only 34.0% ± 0.4% in crystal system classification, close to Gemma-3-1b-it’s 32.9%. This may be related to the imbalanced distribution of crystal system data and the small training batch size, indicating that the model’s understanding of fine geometric features such as crystal symmetry remains limited.

Researchers also selected six typical materials, including Li₇La₃Zr₂O₁₂, GaAs, and BaTiO₃, along with six application scenarios such as solid-state batteries, semiconductors, and supercapacitors, to analyze the similarity between material and application descriptions. Results showed that the model could identify typical associations such as Li₇La₃Zr₂O₁₂ with solid-state batteries, GaAs with semiconductors, and RuO₂ with supercapacitors, indicating that its representations contain certain knowledge of material functions and applications.

In the zero-shot inverse design experiment, researchers reversed the direction of traditional property prediction, retrieving candidate materials based on natural language descriptions of target performance. The experiments involved five properties: band gap, density, convex hull energy difference, scintillation decay length, and total magnetization. For each query, the model retrieved the top 30 most similar materials and measured the difference between their actual properties and target values using Mean Absolute Error (MAE).

Results showed that CLaC-Synthetic outperformed the random baseline in retrieval across all five properties. Specifically, the MAE for band gap retrieval was 0.641 eV, lower than the 1.185 eV for random retrieval; for density retrieval, the MAEs were 0.799 g/cm³ and 2.148 g/cm³, respectively; for scintillation decay length retrieval, the MAEs were 1.033 cm and 2.697 cm, respectively. In contrast, CLaC-Paper’s overall performance was close to random selection, indicating that synthetic narratives generated based on computed properties are more effective than directly extracted academic literature text in establishing the correspondence between language and material properties.

Beyond precise numerical values, the study also tested qualitative descriptions such as "high band gap, low density, strong magnetism." The results revealed that the property distribution of materials retrieved by CLaC-Synthetic varies with query semantics. For instance, when shifting from low to high band gap queries, the band gaps of candidate materials increased accordingly. In contrast, CLaC-Paper struggled to form such distinctions, indicating that synthetic narratives help the model understand the relationship between qualitative descriptions and actual physical properties.

In application validation, researchers used solid-state battery electrolyte screening as an example, retrieving candidates based on lithium-containing materials and high band gap conditions. All top 30 candidates contained lithium, with an average band gap of 3.23 eV. Among them, Li₆PClO₅, ranked fourth, had been identified as a potential candidate in other studies, and Li₄SiO₄, ranked ninth, had previously been fabricated as a solid electrolyte thin film. However, since the training data did not include ionic conductivity information, current results cannot prove that these materials meet all performance requirements for solid electrolytes.

To analyze how the model understands numerical information, the study further examined property continuity in the text embedding space. The results showed that vector similarity in CLaC-Synthetic gradually decreases as the difference in property values increases. For example, using a 2.0 eV band gap as a reference, when the query value increased to 4.0 eV, cosine similarity dropped from 1.0 to approximately 0.65, indicating that the model can express the magnitude relationships of physical quantities. Specifically, similarity decreased more rapidly for density and scintillation decay length, corresponding to higher retrieval accuracy, while changes in convex hull energy difference were relatively gradual, leading to larger retrieval errors. In contrast, neither CLaC-Paper nor Galactica-125M (without cross-modal alignment) exhibited clear continuous numerical relationships.

Finally, researchers used UMAP to visualize the latent spaces of crystal structures and text representations. Although the two modalities formed relatively independent clusters, their material neighborhood relationships were highly consistent: on average, 57% of the 50 nearest neighbors in the crystal graph space also appeared among the 50 nearest neighbors in the text space, significantly higher than the 0.4% observed in random cases. Additionally, physical properties such as band gap and density exhibited regular distributions, and descriptors like "ceramic," "conductor," and "lithium-ion battery" corresponded to distinct crystal regions, further validating the model's cross-modal representation capability for material properties and functional categories.

Conclusion

The CLaC study demonstrates that the key to multimodal learning for crystalline materials lies not only in data scale but also in establishing accurate correspondences between material properties and natural language. By generating synthetic narratives based on computed properties, CLaC outperformed models trained on literature text in zero-shot material understanding and language-guided retrieval. However, the model still falls short in fine-grained structural tasks such as crystal system identification, struggles to handle complex systems like polycrystals, defective structures, and metal-organic frameworks, and cannot directly generate new crystal structures. Future challenges for advancing these models toward practical material design include improving the accuracy of synthetic data, extending model applicability to complex materials, and integrating language retrieval with crystal generation and physical validation.