Initial commit

2025-11-30 08:30:10 +08:00
commit f0bd18fb4e
824 changed files with 331919 additions and 0 deletions
--- a/skills/molfeat/SKILL.md
+++ b/skills/molfeat/SKILL.md
@@ -0,0 +1,505 @@
+---
+name: molfeat
+description: "Molecular featurization for ML (100+ featurizers). ECFP, MACCS, descriptors, pretrained models (ChemBERTa), convert SMILES to features, for QSAR and molecular ML."
+---
+
+# Molfeat - Molecular Featurization Hub
+
+## Overview
+
+Molfeat is a comprehensive Python library for molecular featurization that unifies 100+ pre-trained embeddings and hand-crafted featurizers. Convert chemical structures (SMILES strings or RDKit molecules) into numerical representations for machine learning tasks including QSAR modeling, virtual screening, similarity searching, and deep learning applications. Features fast parallel processing, scikit-learn compatible transformers, and built-in caching.
+
+## When to Use This Skill
+
+This skill should be used when working with:
+- **Molecular machine learning**: Building QSAR/QSPR models, property prediction
+- **Virtual screening**: Ranking compound libraries for biological activity
+- **Similarity searching**: Finding structurally similar molecules
+- **Chemical space analysis**: Clustering, visualization, dimensionality reduction
+- **Deep learning**: Training neural networks on molecular data
+- **Featurization pipelines**: Converting SMILES to ML-ready representations
+- **Cheminformatics**: Any task requiring molecular feature extraction
+
+## Installation
+
+```bash
+uv pip install molfeat
+
+# With all optional dependencies
+uv pip install "molfeat[all]"
+```
+
+**Optional dependencies for specific featurizers:**
+- `molfeat[dgl]` - GNN models (GIN variants)
+- `molfeat[graphormer]` - Graphormer models
+- `molfeat[transformer]` - ChemBERTa, ChemGPT, MolT5
+- `molfeat[fcd]` - FCD descriptors
+- `molfeat[map4]` - MAP4 fingerprints
+
+## Core Concepts
+
+Molfeat organizes featurization into three hierarchical classes:
+
+### 1. Calculators (`molfeat.calc`)
+
+Callable objects that convert individual molecules into feature vectors. Accept RDKit `Chem.Mol` objects or SMILES strings.
+
+**Use calculators for:**
+- Single molecule featurization
+- Custom processing loops
+- Direct feature computation
+
+**Example:**
+```python
+from molfeat.calc import FPCalculator
+
+calc = FPCalculator("ecfp", radius=3, fpSize=2048)
+features = calc("CCO")  # Returns numpy array (2048,)
+```
+
+### 2. Transformers (`molfeat.trans`)
+
+Scikit-learn compatible transformers that wrap calculators for batch processing with parallelization.
+
+**Use transformers for:**
+- Batch featurization of molecular datasets
+- Integration with scikit-learn pipelines
+- Parallel processing (automatic CPU utilization)
+
+**Example:**
+```python
+from molfeat.trans import MoleculeTransformer
+from molfeat.calc import FPCalculator
+
+transformer = MoleculeTransformer(FPCalculator("ecfp"), n_jobs=-1)
+features = transformer(smiles_list)  # Parallel processing
+```
+
+### 3. Pretrained Transformers (`molfeat.trans.pretrained`)
+
+Specialized transformers for deep learning models with batched inference and caching.
+
+**Use pretrained transformers for:**
+- State-of-the-art molecular embeddings
+- Transfer learning from large chemical datasets
+- Deep learning feature extraction
+
+**Example:**
+```python
+from molfeat.trans.pretrained import PretrainedMolTransformer
+
+transformer = PretrainedMolTransformer("ChemBERTa-77M-MLM", n_jobs=-1)
+embeddings = transformer(smiles_list)  # Deep learning embeddings
+```
+
+## Quick Start Workflow
+
+### Basic Featurization
+
+```python
+import datamol as dm
+from molfeat.calc import FPCalculator
+from molfeat.trans import MoleculeTransformer
+
+# Load molecular data
+smiles = ["CCO", "CC(=O)O", "c1ccccc1", "CC(C)O"]
+
+# Create calculator and transformer
+calc = FPCalculator("ecfp", radius=3)
+transformer = MoleculeTransformer(calc, n_jobs=-1)
+
+# Featurize molecules
+features = transformer(smiles)
+print(f"Shape: {features.shape}")  # (4, 2048)
+```
+
+### Save and Load Configuration
+
+```python
+# Save featurizer configuration for reproducibility
+transformer.to_state_yaml_file("featurizer_config.yml")
+
+# Reload exact configuration
+loaded = MoleculeTransformer.from_state_yaml_file("featurizer_config.yml")
+```
+
+### Handle Errors Gracefully
+
+```python
+# Process dataset with potentially invalid SMILES
+transformer = MoleculeTransformer(
+    calc,
+    n_jobs=-1,
+    ignore_errors=True,  # Continue on failures
+    verbose=True          # Log error details
+)
+
+features = transformer(smiles_with_errors)
+# Returns None for failed molecules
+```
+
+## Choosing the Right Featurizer
+
+### For Traditional Machine Learning (RF, SVM, XGBoost)
+
+**Start with fingerprints:**
+```python
+# ECFP - Most popular, general-purpose
+FPCalculator("ecfp", radius=3, fpSize=2048)
+
+# MACCS - Fast, good for scaffold hopping
+FPCalculator("maccs")
+
+# MAP4 - Efficient for large-scale screening
+FPCalculator("map4")
+```
+
+**For interpretable models:**
+```python
+# RDKit 2D descriptors (200+ named properties)
+from molfeat.calc import RDKitDescriptors2D
+RDKitDescriptors2D()
+
+# Mordred (1800+ comprehensive descriptors)
+from molfeat.calc import MordredDescriptors
+MordredDescriptors()
+```
+
+**Combine multiple featurizers:**
+```python
+from molfeat.trans import FeatConcat
+
+concat = FeatConcat([
+    FPCalculator("maccs"),      # 167 dimensions
+    FPCalculator("ecfp")         # 2048 dimensions
+])  # Result: 2215-dimensional combined features
+```
+
+### For Deep Learning
+
+**Transformer-based embeddings:**
+```python
+# ChemBERTa - Pre-trained on 77M PubChem compounds
+PretrainedMolTransformer("ChemBERTa-77M-MLM")
+
+# ChemGPT - Autoregressive language model
+PretrainedMolTransformer("ChemGPT-1.2B")
+```
+
+**Graph neural networks:**
+```python
+# GIN models with different pre-training objectives
+PretrainedMolTransformer("gin-supervised-masking")
+PretrainedMolTransformer("gin-supervised-infomax")
+
+# Graphormer for quantum chemistry
+PretrainedMolTransformer("Graphormer-pcqm4mv2")
+```
+
+### For Similarity Searching
+
+```python
+# ECFP - General purpose, most widely used
+FPCalculator("ecfp")
+
+# MACCS - Fast, scaffold-based similarity
+FPCalculator("maccs")
+
+# MAP4 - Efficient for large databases
+FPCalculator("map4")
+
+# USR/USRCAT - 3D shape similarity
+from molfeat.calc import USRDescriptors
+USRDescriptors()
+```
+
+### For Pharmacophore-Based Approaches
+
+```python
+# FCFP - Functional group based
+FPCalculator("fcfp")
+
+# CATS - Pharmacophore pair distributions
+from molfeat.calc import CATSCalculator
+CATSCalculator(mode="2D")
+
+# Gobbi - Explicit pharmacophore features
+FPCalculator("gobbi2D")
+```
+
+## Common Workflows
+
+### Building a QSAR Model
+
+```python
+from molfeat.trans import MoleculeTransformer
+from molfeat.calc import FPCalculator
+from sklearn.ensemble import RandomForestRegressor
+from sklearn.model_selection import cross_val_score
+
+# Featurize molecules
+transformer = MoleculeTransformer(FPCalculator("ecfp"), n_jobs=-1)
+X = transformer(smiles_train)
+
+# Train model
+model = RandomForestRegressor(n_estimators=100)
+scores = cross_val_score(model, X, y_train, cv=5)
+print(f"R² = {scores.mean():.3f}")
+
+# Save configuration for deployment
+transformer.to_state_yaml_file("production_featurizer.yml")
+```
+
+### Virtual Screening Pipeline
+
+```python
+from sklearn.ensemble import RandomForestClassifier
+
+# Train on known actives/inactives
+transformer = MoleculeTransformer(FPCalculator("ecfp"), n_jobs=-1)
+X_train = transformer(train_smiles)
+clf = RandomForestClassifier(n_estimators=500)
+clf.fit(X_train, train_labels)
+
+# Screen large library
+X_screen = transformer(screening_library)  # e.g., 1M compounds
+predictions = clf.predict_proba(X_screen)[:, 1]
+
+# Rank and select top hits
+top_indices = predictions.argsort()[::-1][:1000]
+top_hits = [screening_library[i] for i in top_indices]
+```
+
+### Similarity Search
+
+```python
+from sklearn.metrics.pairwise import cosine_similarity
+
+# Query molecule
+calc = FPCalculator("ecfp")
+query_fp = calc(query_smiles).reshape(1, -1)
+
+# Database fingerprints
+transformer = MoleculeTransformer(calc, n_jobs=-1)
+database_fps = transformer(database_smiles)
+
+# Compute similarity
+similarities = cosine_similarity(query_fp, database_fps)[0]
+top_similar = similarities.argsort()[-10:][::-1]
+```
+
+### Scikit-learn Pipeline Integration
+
+```python
+from sklearn.pipeline import Pipeline
+from sklearn.ensemble import RandomForestClassifier
+
+# Create end-to-end pipeline
+pipeline = Pipeline([
+    ('featurizer', MoleculeTransformer(FPCalculator("ecfp"), n_jobs=-1)),
+    ('classifier', RandomForestClassifier(n_estimators=100))
+])
+
+# Train and predict directly on SMILES
+pipeline.fit(smiles_train, y_train)
+predictions = pipeline.predict(smiles_test)
+```
+
+### Comparing Multiple Featurizers
+
+```python
+featurizers = {
+    'ECFP': FPCalculator("ecfp"),
+    'MACCS': FPCalculator("maccs"),
+    'Descriptors': RDKitDescriptors2D(),
+    'ChemBERTa': PretrainedMolTransformer("ChemBERTa-77M-MLM")
+}
+
+results = {}
+for name, feat in featurizers.items():
+    transformer = MoleculeTransformer(feat, n_jobs=-1)
+    X = transformer(smiles)
+    # Evaluate with your ML model
+    score = evaluate_model(X, y)
+    results[name] = score
+```
+
+## Discovering Available Featurizers
+
+Use the ModelStore to explore all available featurizers:
+
+```python
+from molfeat.store.modelstore import ModelStore
+
+store = ModelStore()
+
+# List all available models
+all_models = store.available_models
+print(f"Total featurizers: {len(all_models)}")
+
+# Search for specific models
+chemberta_models = store.search(name="ChemBERTa")
+for model in chemberta_models:
+    print(f"- {model.name}: {model.description}")
+
+# Get usage information
+model_card = store.search(name="ChemBERTa-77M-MLM")[0]
+model_card.usage()  # Display usage examples
+
+# Load model
+transformer = store.load("ChemBERTa-77M-MLM")
+```
+
+## Advanced Features
+
+### Custom Preprocessing
+
+```python
+class CustomTransformer(MoleculeTransformer):
+    def preprocess(self, mol):
+        """Custom preprocessing pipeline"""
+        if isinstance(mol, str):
+            mol = dm.to_mol(mol)
+        mol = dm.standardize_mol(mol)
+        mol = dm.remove_salts(mol)
+        return mol
+
+transformer = CustomTransformer(FPCalculator("ecfp"), n_jobs=-1)
+```
+
+### Batch Processing Large Datasets
+
+```python
+def featurize_in_chunks(smiles_list, transformer, chunk_size=10000):
+    """Process large datasets in chunks to manage memory"""
+    all_features = []
+    for i in range(0, len(smiles_list), chunk_size):
+        chunk = smiles_list[i:i+chunk_size]
+        features = transformer(chunk)
+        all_features.append(features)
+    return np.vstack(all_features)
+```
+
+### Caching Expensive Embeddings
+
+```python
+import pickle
+
+cache_file = "embeddings_cache.pkl"
+transformer = PretrainedMolTransformer("ChemBERTa-77M-MLM", n_jobs=-1)
+
+try:
+    with open(cache_file, "rb") as f:
+        embeddings = pickle.load(f)
+except FileNotFoundError:
+    embeddings = transformer(smiles_list)
+    with open(cache_file, "wb") as f:
+        pickle.dump(embeddings, f)
+```
+
+## Performance Tips
+
+1. **Use parallelization**: Set `n_jobs=-1` to utilize all CPU cores
+2. **Batch processing**: Process multiple molecules at once instead of loops
+3. **Choose appropriate featurizers**: Fingerprints are faster than deep learning models
+4. **Cache pretrained models**: Leverage built-in caching for repeated use
+5. **Use float32**: Set `dtype=np.float32` when precision allows
+6. **Handle errors efficiently**: Use `ignore_errors=True` for large datasets
+
+## Common Featurizers Reference
+
+**Quick reference for frequently used featurizers:**
+
+| Featurizer | Type | Dimensions | Speed | Use Case |
+|------------|------|------------|-------|----------|
+| `ecfp` | Fingerprint | 2048 | Fast | General purpose |
+| `maccs` | Fingerprint | 167 | Very fast | Scaffold similarity |
+| `desc2D` | Descriptors | 200+ | Fast | Interpretable models |
+| `mordred` | Descriptors | 1800+ | Medium | Comprehensive features |
+| `map4` | Fingerprint | 1024 | Fast | Large-scale screening |
+| `ChemBERTa-77M-MLM` | Deep learning | 768 | Slow* | Transfer learning |
+| `gin-supervised-masking` | GNN | Variable | Slow* | Graph-based models |
+
+*First run is slow; subsequent runs benefit from caching
+
+## Resources
+
+This skill includes comprehensive reference documentation:
+
+### references/api_reference.md
+Complete API documentation covering:
+- `molfeat.calc` - All calculator classes and parameters
+- `molfeat.trans` - Transformer classes and methods
+- `molfeat.store` - ModelStore usage
+- Common patterns and integration examples
+- Performance optimization tips
+
+**When to load:** Reference when implementing specific calculators, understanding transformer parameters, or integrating with scikit-learn/PyTorch.
+
+### references/available_featurizers.md
+Comprehensive catalog of all 100+ featurizers organized by category:
+- Transformer-based language models (ChemBERTa, ChemGPT)
+- Graph neural networks (GIN, Graphormer)
+- Molecular descriptors (RDKit, Mordred)
+- Fingerprints (ECFP, MACCS, MAP4, and 15+ others)
+- Pharmacophore descriptors (CATS, Gobbi)
+- Shape descriptors (USR, ElectroShape)
+- Scaffold-based descriptors
+
+**When to load:** Reference when selecting the optimal featurizer for a specific task, exploring available options, or understanding featurizer characteristics.
+
+**Search tip:** Use grep to find specific featurizer types:
+```bash
+grep -i "chembert" references/available_featurizers.md
+grep -i "pharmacophore" references/available_featurizers.md
+```
+
+### references/examples.md
+Practical code examples for common scenarios:
+- Installation and quick start
+- Calculator and transformer examples
+- Pretrained model usage
+- Scikit-learn and PyTorch integration
+- Virtual screening workflows
+- QSAR model building
+- Similarity searching
+- Troubleshooting and best practices
+
+**When to load:** Reference when implementing specific workflows, troubleshooting issues, or learning molfeat patterns.
+
+## Troubleshooting
+
+### Invalid Molecules
+Enable error handling to skip invalid SMILES:
+```python
+transformer = MoleculeTransformer(
+    calc,
+    ignore_errors=True,
+    verbose=True
+)
+```
+
+### Memory Issues with Large Datasets
+Process in chunks or use streaming approaches for datasets > 100K molecules.
+
+### Pretrained Model Dependencies
+Some models require additional packages. Install specific extras:
+```bash
+uv pip install "molfeat[transformer]"  # For ChemBERTa/ChemGPT
+uv pip install "molfeat[dgl]"          # For GIN models
+```
+
+### Reproducibility
+Save exact configurations and document versions:
+```python
+transformer.to_state_yaml_file("config.yml")
+import molfeat
+print(f"molfeat version: {molfeat.__version__}")
+```
+
+## Additional Resources
+
+- **Official Documentation**: https://molfeat-docs.datamol.io/
+- **GitHub Repository**: https://github.com/datamol-io/molfeat
+- **PyPI Package**: https://pypi.org/project/molfeat/
+- **Tutorial**: https://portal.valencelabs.com/datamol/post/types-of-featurizers-b1e8HHrbFMkbun6