Recent searches

No recent searches

Search options

Only available when logged in.
sigmoid.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
A social space for people researching, working with, or just interested in AI!

Administered by:

Server stats:

592
active users

sigmoid.social: AboutProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.4.1

Sanket Vaibhav Mehta

New preprint 🚨

DSI++: Updating Transformer Memory with New Documents

Q: "Can you add new documents to DSI??" was the big question many people had when DSI first came out.
A: Turns out you actually can!

arxiv.org/abs/2212.09744

(1/n)

arXiv.orgDSI++: Updating Transformer Memory with New DocumentsDifferentiable Search Indices (DSIs) encode a corpus of documents in model parameters and use the same model to answer user queries directly. Despite the strong performance of DSI models, deploying them in situations where the corpus changes over time is computationally expensive because reindexing the corpus requires re-training the model. In this work, we introduce DSI++, a continual learning challenge for DSI to incrementally index new documents while being able to answer queries related to both previously and newly indexed documents. Across different model scales and document identifier representations, we show that continual indexing of new documents leads to considerable forgetting of previously indexed documents. We also hypothesize and verify that the model experiences forgetting events during training, leading to unstable learning. To mitigate these issues, we investigate two approaches. The first focuses on modifying the training dynamics. Flatter minima implicitly alleviate forgetting, so we optimize for flatter loss basins and show that the model stably memorizes more documents ($+12\%$). Next, we introduce a generative memory to sample pseudo-queries for documents and supplement them during continual indexing to prevent forgetting for the retrieval task. Extensive experiments on novel continual indexing benchmarks based on Natural Questions (NQ) and MS MARCO demonstrate that our proposed solution mitigates forgetting significantly. Concretely, it improves the average Hits@10 by $+21.1\%$ over competitive baselines for NQ and requires $6$ times fewer model updates compared to re-training the DSI model for incrementally indexing five corpora in a sequence.
··Web
7boosts·7favorites
Sanket Vaibhav Mehta

Q: Why DSI++?
A: Deploying the DSI model in situations where the corpus changes over time is computationally expensive because reindexing the corpus requires re-training the model

(3/n)

Sanket Vaibhav Mehta

Q: What is DSI++?
A: DSI++ (DSI + new documents) 👉 a step towards incrementally indexing new documents in the DSI model by being computationally efficient and maintaining the ability to answer user queries related to both previously and newly indexed documents

(4/n)

Sanket Vaibhav Mehta

Q: What are the challenges for enabling DSI++?
A: (a) Catastrophic forgetting during continual indexing 👉 a common phenomenon in neural networks wherein learning of the new documents interferes with the previously memorized documents

(5/n)

Sanket Vaibhav Mehta

…(b) Implicit forgetting during memorization 👉 we observe a significant number of documents (~88%) experience forgetting events (when prediction for an individual document goes from correct docid to incorrect one) after they have been memorized

(6/n)

Sanket Vaibhav Mehta

Q: How severe is the forgetting of the originally indexed documents? How does the updated DSI model perform on newly indexed documents? How do different docid representation strategies affect forgetting? How does the DSI model scale affect forgetting?
A: Systematic study 👇

(7/n)

Sanket Vaibhav Mehta

Q: How to reduce forgetting during memorization?
A: Flatter minima are shown to reduce forgetting...we explicitly optimize for flatter minima using Sharpness-Aware Minimization (SAM) procedure...see our results 👇
SAM paper 👉 openreview.net/forum?id=6Tm1mp

(8/n)

Sanket Vaibhav Mehta

Q: How to alleviate forgetting during continual indexing?
A: Generative memory 👉 a parametric model to generate queries for documents…use it for sparse experience replay (ER) for already indexed documents...enables continual semi-supervised learning for new documents...👇

(9/n)

Sanket Vaibhav Mehta

Q: Does our proposed approach for DSI++ generalize to different datasets?
A: We show convincing results across two DSI++ benchmarks, constructed from publicly available datasets – Natural Questions (NQ) 👆 and MS MARCO 👇

(10/n)

Sanket Vaibhav Mehta

Q: Want to know more?
A: Look at our paper for the effectiveness of the generative memory with the scale of a corpus (8.9M MS MARCO passages), sparsity of ER on forgetting, analysis around incremental index construction time

DSI++ paper 👉 arxiv.org/abs/2212.09744

(11/n)

Sanket Vaibhav Mehta

I am deeply grateful to my incredible co-authors at and for their invaluable assistance!

(12/n)

TrendingLive feeds
About