Skip to content

Commit

Permalink
notes updates
Browse files Browse the repository at this point in the history
  • Loading branch information
@csinva
csinva committed Jan 23, 2025
1 parent 0195ddf commit 30c698b
Show file tree
Hide file tree
Showing 3 changed files with 73 additions and 36 deletions.
79 changes: 47 additions & 32 deletions _includes/01_research.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,19 +21,24 @@ <h2 style="text-align: center; margin-top: -150px;"> Research</h2>
are interested in interning at MSR, feel free to reach out over email :)</div>

<div class="research_box"><strong>🔎
Interpretability.</strong> I'm interested in <a href="https://arxiv.org/abs/2402.01761">rethinking
interpretability</a> in the context of LLMs
Interpretability methods,</strong> especially <a href="https://arxiv.org/abs/2402.01761">LLM
interpretability</a>.
<br>
<br>
<a href="https://www.nature.com/articles/s41467-023-43713-1">augmented imodels</a> - use LLMs to build a
transparent model<br>
<!-- <a href="https://arxiv.org/abs/2310.14034">tree prompting</a> - improve black-box few-shot text classification -->
<!-- with decision trees<br> -->
<a href="https://arxiv.org/abs/2311.02262">attention steering</a> - mechanistically guide LLMs by
emphasizing specific input
spans<br>
<a href="http://proceedings.mlr.press/v119/rieger20a.html">explanation penalization</a> - regularize
explanations to align models with prior knowledge<br>
<a href="https://proceedings.neurips.cc/paper/2021/file/acaa23f71f963e96c8847585e71352d6-Paper.pdf">adaptive
wavelet distillation</a> - replace neural nets with simple, performant wavelet models
wavelet distillation</a> - replace neural nets with transparent wavelet models
</div>

<div class="research_box">
<!-- <div class="research_box">
<strong>🚗 LLM steering. </strong>Interpretability tools can provide ways to better guide and use LLMs (without
needing gradients!)
Expand All @@ -46,43 +51,49 @@ <h2 style="text-align: center; margin-top: -150px;"> Research</h2>
spans<br>
<a href="https://arxiv.org/abs/2210.01848">interpretable autoprompting</a> - automatically find fluent
natural-language prompts<br>
</div>
</div> -->


<div class="research_box">

<strong>🧠 Neuroscience. </strong> Since joining MSR, I've been focused on leveraging LLM interpretability
to understand how the human brain represents language (using fMRI in collaboration with the <a
href="https://www.cs.utexas.edu/~huth/index.html">Huth lab</a> at UT Austin).
<strong>🧠 Semantic brain mapping, </strong> mostly using fMRI responses to language.
<!-- Since joining MSR, I've been focused on leveraging LLM interpretability -->
<!-- to understand how the human brain represents language (using fMRI in collaboration with the <a -->
<!-- href="https://www.cs.utexas.edu/~huth/index.html">Huth lab</a> at UT Austin). -->
<br>
<br>
<a href="https://arxiv.org/abs/2410.00812">explanation-mediated validation</a> - build and test fMRI
<a href="https://arxiv.org/abs/2410.00812">explanation-mediated validation</a> - test fMRI
explanations using LLM-generated stimuli<br>
<a href="https://arxiv.org/abs/2405.16714">qa embeddings</a> - build interpretable fMRI encoding models by
<a href="https://arxiv.org/abs/2405.16714">qa embeddings</a> - predict fMRI language responses by
asking yes/no questions to LLMs<br>
<a href="https://arxiv.org/abs/2305.09863">summarize &amp; score explanations</a> - generate natural-language
explanations of fMRI encoding models
explanations of fMRI encoding models<br>
</div>


<div class="research_box"><strong>💊
Healthcare. </strong>I'm also actively working on how we can improve clinical decision instruments by using
the information contained across various sources in the medical literature (in collaboration with <a
href="https://profiles.ucsf.edu/aaron.kornblith">Aaron Kornblith</a> at UCSF and the MSR <a
href="https://www.microsoft.com/en-us/research/group/real-world-evidence/">Health Futures team</a>).
Clinical decision rules, </strong>can we improve them with data?
<!-- I'm also actively working on how we can improve clinical decision -->
<!-- instruments by using -->
<!-- the information contained across various sources in the medical literature (in collaboration with <a -->
<!-- href="https://profiles.ucsf.edu/aaron.kornblith">Aaron Kornblith</a> at UCSF and the MSR <a -->
<!-- href="https://www.microsoft.com/en-us/research/group/real-world-evidence/">Health Futures team</a>). -->
<br>
<br>
<a href="https://arxiv.org/pdf/2201.11931">greedy tree sums</a> - build accurate, compact tree-based clinical
models<br>
<a href="https://arxiv.org/abs/2306.00024">clinical self-verification</a> - self-verification improves
performance and interpretability of clinical information extraction<br>
<a href="https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000076">clinical rule
vetting</a> - stress testing a clinical decision instrument performance for intra-abdominal injury

vetting</a> - stress testing a clinical decision instrument performance for intra-abdominal injury<br>
</div>

<div style="width: 100%;padding: 8px;margin-bottom: 20px; text-align:center; font-size: large;">
Across these areas, I'm interested in decision trees and how we can build flexible but accurate transparent
models. I put a lot of my code into the <a href="https://github.com/csinva/imodels">imodels</a> and <a
href="https://github.com/csinva/imodelsx">imodelsX</a> packages.</div>
<!-- Across these areas, I'm interested in decision trees and how we can build flexible but accurate transparent -->
<!-- models. -->
Note: I put a lot of my code into the <a href="https://github.com/csinva/imodels">imodels</a> and <a
href="https://github.com/csinva/imodelsx">imodelsX</a> packages.
</div>
</div>

<hr>
Expand Down Expand Up @@ -153,6 +164,18 @@ <h2 style="text-align: center; margin-top: -150px;"> Research</h2>
</tr>
</thead>
<tbody>
<tr>
<td class="center">'25</td>
<td>Vector-ICL: In-context Learning with Continuous Vector Representations
</td>
<td>zhuang et al.</td>
<td class="med">🔎🌀</td>
<td class="center"><a href="https://arxiv.org/abs/2410.05629">iclr</a></td>
<td class="big"><a href="https://github.com/EvanZhuang/vector-icl"><i class="fa fa-github fa-fw"></i></a>
</td>
<td class="med">
</td>
</tr>
<tr>
<td class="center">'24</td>
<td>Interpretable Language Modeling via Induction-head Ngram Models
Expand All @@ -175,6 +198,8 @@ <h2 style="text-align: center; margin-top: -150px;"> Research</h2>
<td class="big"><a href="https://github.com/microsoft/automated-explanations"><i
class="fa fa-github fa-fw"></i></a></td>
<td class="med">
<a href="https://docs.google.com/presentation/d/1bFZZ8-OwwNxN3DjPdyPFKuP16yyYTP9DlaTzKOaYVWI/"><i
class="fa fa-desktop fa-fw"></i></a>
</td>
</tr>
<tr>
Expand All @@ -187,6 +212,8 @@ <h2 style="text-align: center; margin-top: -150px;"> Research</h2>
<td class="big"><a href="https://github.com/csinva/interpretable-embeddings"><i
class="fa fa-github fa-fw"></i></a></td>
<td class="med">
<a href="https://docs.google.com/presentation/d/1bFZZ8-OwwNxN3DjPdyPFKuP16yyYTP9DlaTzKOaYVWI/"><i
class="fa fa-desktop fa-fw"></i></a>
</td>
</tr>
<tr>
Expand All @@ -200,18 +227,6 @@ <h2 style="text-align: center; margin-top: -150px;"> Research</h2>
<td class="med">
</td>
</tr>
<tr>
<td class="center">'24</td>
<td>Vector-ICL: In-context Learning with Continuous Vector Representations
</td>
<td>zhuang et al.</td>
<td class="med">🔎🌀</td>
<td class="center"><a href="https://arxiv.org/abs/2410.05629">arxiv</a></td>
<td class="big"><a href="https://github.com/EvanZhuang/vector-icl"><i class="fa fa-github fa-fw"></i></a>
</td>
<td class="med">
</td>
</tr>
<tr>
<td class="center">'24</td>
<td>Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning
Expand Down
8 changes: 7 additions & 1 deletion _notes/research_ovws/ovw_interp.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,12 @@ For an implementation of many of these models, see the python [imodels package](
- longitudinal data, survival curves

- misc

- On the Power of Decision Trees in Auto-Regressive Language Modeling ([gan, galanti, poggio, malach, 2024](https://arxiv.org/pdf/2409.19150))
- get token word embeddings
- compute exp. weighted avg of embeddings (upweights most recent tokens)
- predicts next embedding with XGBoost (regression loss) then finds closest token

- counterfactuals
- [Counterfactual Explanations for Oblique Decision Trees: Exact, Efficient Algorithms](https://arxiv.org/abs/2103.01096) (2021)
- [Optimal Counterfactual Explanations in Tree Ensembles](https://arxiv.org/abs/2106.06631)
Expand All @@ -322,7 +328,7 @@ For an implementation of many of these models, see the python [imodels package](
1. feature-level: monotonicity, attribute costs, hierarchy/interaction, fairness, privacy
2. structure-level - e.g. minimize #nodes
3. instance-level - must (cannot) link, robust predictions

- Analysis of Boolean functions ([wiki](https://en.wikipedia.org/wiki/Analysis_of_Boolean_functions))

- Every real-valued function $f:\{-1,1\}^n \rightarrow \mathbb{R}$ has a unique expansion as a multilinear polynomial:
Expand Down
22 changes: 19 additions & 3 deletions _notes/research_ovws/ovw_llms.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1283,7 +1283,7 @@ mixture of experts models have become popular because of the need for (1) fast s
- Ravel: Evaluating Interpretability Methods on Disentangling Language Model Representations ([huang, wu, potts, geva, & geiger, 2024](https://arxiv.org/pdf/2402.17700v1.pdf))


## directly learning algorithms / in-context
## directly learning algorithms

- Empirical results
- FunSearch: Mathematical discoveries from program search with LLMs ([deepmind, 2023](https://www.nature.com/articles/s41586-023-06924-6))
Expand All @@ -1294,6 +1294,10 @@ mixture of experts models have become popular because of the need for (1) fast s
- Alphafold
- Accurate proteome-wide missense variant effect prediction with AlphaMissense ([deepmind, 2023](https://www.science.org/doi/full/10.1126/science.adg7492)) - predict effects of varying single-amino acid changes
- Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZero ([schut...hessabis, paquet, & been kim, 2023](https://arxiv.org/abs/2310.16410))
- Learning a Decision Tree Algorithm with Transformers ([zhuang...gao, 2024](https://arxiv.org/abs/2402.03774))

## in-context learning

- What Can Transformers Learn In-Context? A Case Study of Simple Function Classes ([garg, tsipras, liang, & valiant, 2022](https://arxiv.org/abs/2208.01066)) - models can succesfully metalearn functions like OLS
- e.g. during training, learn inputs-outputs from different linear functions
- during testing, have to predict outputs for inputs from a different linear function
Expand Down Expand Up @@ -1328,6 +1332,12 @@ mixture of experts models have become popular because of the need for (1) fast s
- Transformers are Universal In-context Learners ([furuya...peyre, 2024](https://arxiv.org/abs/2408.01367)) - mathetmatically show that transformers are universal and can approximate continuous in-context mappings to arbitrary precision
- Limitations
- Faith and Fate: Limits of Transformers on Compositionality ([dziri...choi, 2023](https://arxiv.org/abs/2305.18654)) - LLMs can't (easily) be trained well for multiplication (and similar tasks)
- ICLR: In-Context Learning of Representations ([park...wattenberg, tanaka, 2024](https://arxiv.org/abs/2501.00070)) - showing pairs of words sampled from a graph can make the embeddings of those words match the structure of that graph
- Label Words are Anchors: An Information Flow Perspective for
Understanding In-Context Learning ([wang...sun, 2023](https://aclanthology.org/2023.emnlp-main.609.pdf))
- Correlation and Navigation in the Vocabulary Key Representation Space of Language Models ([peng...shang, 2024](https://arxiv.org/abs/2410.02284)) - some tokens are correlated in embedding space and wrong next-token completions can be highly ranked if their embeddings are correlated with correct ones
- as we sample tokens in context, we get more diverse completions, skipping nearby wrong next tokens


## cool tasks

Expand Down Expand Up @@ -1483,7 +1493,8 @@ mixture of experts models have become popular because of the need for (1) fast s
- TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data ([yin, neubig, ..., riedel, 2020](https://www.semanticscholar.org/paper/TaBERT%3A-Pretraining-for-Joint-Understanding-of-and-Yin-Neubig/a5b1d1cab073cb746a990b37d42dc7b67763f881))

- classification / predictions
- TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second ([hollman, ..., hutter, 2022](https://arxiv.org/abs/2207.01848))
- TabPFN v2: Accurate predictions on small data with a tabular foundation model ([hollman....hutter, 2025](https://www.nature.com/articles/s41586-024-08328-6))
- TabPFN v1: A Transformer That Solves Small Tabular Classification Problems in a Second ([hollman, ..., hutter, 2022](https://arxiv.org/abs/2207.01848))
- transformer takes in train + test dataset then outputs predictions
- each row (data example) is treated as a token and test points attend only to training t
- takes fixed-size 100 columns, with zero-padded columns at the end (during training, randomly subsample columns)
Expand All @@ -1494,7 +1505,7 @@ mixture of experts models have become popular because of the need for (1) fast s
- Language models are weak learners ([manikandan, jian, & kolter, 2023](https://arxiv.org/abs/2306.14101)) - use prompted LLMs as weak learners in boosting algorithm for tabular data
- TabRet: Pre-training Transformer-based Tabular Models for Unseen Columns ([onishi...hayashi, 2023](https://arxiv.org/abs/2303.15747))
- AnyPredict: A Universal Tabular Prediction System Based on LLMs https://openreview.net/forum?id=icuV4s8f2c - converting tabular data into machine-understandable prompts and fine-tuning LLMs to perform accurate predictions

- interpretability
- InterpreTabNet: Enhancing Interpretability of Tabular Data Using Deep Generative Models and LLM ([si...krishnan, 2023](https://openreview.net/pdf?id=kzR5Cj5blw)) - make attention sparse and describe it with GPT4

Expand All @@ -1519,6 +1530,11 @@ mixture of experts models have become popular because of the need for (1) fast s
- Embeddings for Tabular Data: A Survey ([singh & bedathur, 2023](https://arxiv.org/abs/2302.11777))
- Deep neural networks and tabular data: A survey ([borisov et al. 2022]()) - mostly compares performance on standard tasks (e.g. classification)

## education

- Towards Responsible Development of Generative AI for Education: An Evaluation-Driven Approach ([jurenka…ibrahim, 2024](https://storage.googleapis.com/deepmind-media/LearnLM/LearnLM_paper.pdf))
- seven diverse educational benchmark
- The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Response?s to Long-Form Input ([jacovi…das, 2025](https://arxiv.org/abs/2501.03200)) - benchmark evaluates whether responses are consistent with a provided document as context

## llm limitations / perspectives

Expand Down

0 comments on commit 30c698b

Please sign in to comment.