Q&A With Predictive Coding Guru, Maura R. Grossman, Esq.
Can you tell us a little about your practice and your interest in predictive coding?
After a prior career as a clinical psychologist, I joined Wachtell Lipton as a litigator in 1999, and in 2007, when I was promoted to counsel, my practice shifted exclusively to advising lawyers and clients on legal, technical, and strategic issues involving electronic discovery and information management, both domestically and abroad.
I became interested in technology-assisted review (“TAR”) in the 2007/2008 time frame, when I sought to address the fact that Wachtell Lipton had few associates to devote to document review, and contract attorney review was costly, time-consuming, and generally of poor quality. At about the same time, I crossed paths with Jason R. Baron and got involved in the TREC Legal Track.
What are a few of the biggest predictive coding myths?
There are so many, it’s hard to limit myself to only a few! Here are my nominations for the top ten, in no particular order:
Myth #1: TAR is the same thing as clustering, concept search, “find similar,” or any number of other early case assessment tools.
Myth #2: Seed or training sets must always be random.
Myth #3: Seed or training sets must always be selected and reviewed by senior partners.
Myth #4: Thousands of documents must be reviewed as a prerequisite to employing TAR, therefore, it is not suitable for smaller matters.
Myth #5: TAR is more susceptible to reviewer error than the “traditional approach.”
Myth #6: One should cull with keywords prior to employing TAR.
Myth #7: TAR does not work for short documents, spreadsheets, foreign language documents, or OCR’d documents.
Myth #8: Tar finds “easy” documents at the expense of “hot” documents.
Myth #9: If one adds new custodians to the collection, one must always retrain the system.
Myth #10: Small changes to the seed or training set can cause large changes in the outcome, for example, documents that were previously tagged as highly relevant can become non-relevant.
The bottom line is that your readers should challenge commonly held (and promoted) assumptions that lack empirical support.
Are all predictive coding tools the same? If not, then what should legal departments look for when selecting a predictive coding tool?
Not at all, and neither are all manual reviews. It is important to ask service providers the right questions to understand what you are getting. For example, some TAR tools employ supervised or active machine learning, which require the construction of a “training set” of documents to teach the classifier to distinguish between responsive and non-responsive documents. Supervised learning methods are generally more static, while active learning methods involve more interaction with the tool and more iteration. Knowledge engineering approaches (a.k.a. “rule-based” methods) involve the construction of linguistic and other models that replicate the way that humans think about complex problems. Both approaches can be effective when properly employed and validated. At this time, only active machine learning and rule-based approaches have been shown to be effective for technology-assisted review. Service providers should be prepared to tell their clients what is “under the hood.”
What is the number one mistake practitioners should avoid when using these tools?
Not employing proper validation protocols, which are essential to a defensible process. There is widespread misunderstanding of statistics and what they can and cannot tell us. For example, many service providers report that their tools achieve 99% accuracy. Accuracy is the fraction of documents that are correctly coded by a search or review effort. While accuracy is commonly advanced as evidence of an effective search or review effort, it can be misleading because it is heavily influenced by prevalence, or the number of responsive documents in the collection. Consider, for example, a document collection containing one million documents, of which ten thousand (or 1%) are relevant. A search or review effort that identified 100% of the documents as non-relevant, and therefore, found none of the relevant documents, would have 99% accuracy, belying the failure of that search or review effort to identify a single relevant document.
What do you see as the key issues that will confront practitioners who wish to use predictive coding in the near-term?
There are several issues that will be played out in the courts and in practice over the next few years. They include: (1) How does one know if the proposed TAR tool will work (or did work) as advertised?; (2) Must seed or training sets be disclosed, and why?; (3) Must documents coded as non-relevant be disclosed, and why?; (4) Should TAR be held to a higher standard of validation than manual review?; and (5) What cost and effort is justified for the purposes of validation? How does one ensure that the cost of validation does not obliterate the savings achieved by using TAR?
What have you been up to lately?
In an effort to bring order to chaos by introducing a common framework and set of definitions for use by the bar, bench, and vendor community, Gordon V. Cormack and I recently prepared a glossary on technology-assisted review that is available for free download at: http://cormack.uwaterloo.ca/targlossary. We hope that your readers will send us their comments on our definitions and additional terms for inclusion in the next version of the glossary.
Maura R. Grossman, counsel at Wachtell, Lipton, Rosen & Katz, is a well-known e-discovery lawyer and recognized expert in technology-assisted review. Her work was cited in the landmark 2012 case, Da Silva Moore v. Publicis Group (S.D.N.Y. 2012).
Q&A With Predictive Coding Guru, Maura R. Grossman, Esq.
BY MATTHEW NELSON ON NOVEMBER 13TH, 2012
Google +10
Can you tell us a little about your practice and your interest in predictive coding?After a prior career as a clinical psychologist, I joined Wachtell Lipton as a litigator in 1999, and in 2007, when I was promoted to counsel, my practice shifted exclusively to advising lawyers and clients on legal, technical, and strategic issues involving electronic discovery and information management, both domestically and abroad.
I became interested in technology-assisted review (“TAR”) in the 2007/2008 time frame, when I sought to address the fact that Wachtell Lipton had few associates to devote to document review, and contract attorney review was costly, time-consuming, and generally of poor quality. At about the same time, I crossed paths with Jason R. Baron and got involved in the TREC Legal Track.
What are a few of the biggest predictive coding myths?
There are so many, it’s hard to limit myself to only a few! Here are my nominations for the top ten, in no particular order:
Myth #1: TAR is the same thing as clustering, concept search, “find similar,” or any number of other early case assessment tools.
Myth #2: Seed or training sets must always be random.
Myth #3: Seed or training sets must always be selected and reviewed by senior partners.
Myth #4: Thousands of documents must be reviewed as a prerequisite to employing TAR, therefore, it is not suitable for smaller matters.
Myth #5: TAR is more susceptible to reviewer error than the “traditional approach.”
Myth #6: One should cull with keywords prior to employing TAR.
Myth #7: TAR does not work for short documents, spreadsheets, foreign language documents, or OCR’d documents.
Myth #8: Tar finds “easy” documents at the expense of “hot” documents.
Myth #9: If one adds new custodians to the collection, one must always retrain the system.
Myth #10: Small changes to the seed or training set can cause large changes in the outcome, for example, documents that were previously tagged as highly relevant can become non-relevant.
The bottom line is that your readers should challenge commonly held (and promoted) assumptions that lack empirical support.
Are all predictive coding tools the same? If not, then what should legal departments look for when selecting a predictive coding tool?
Not at all, and neither are all manual reviews. It is important to ask service providers the right questions to understand what you are getting. For example, some TAR tools employ supervised or active machine learning, which require the construction of a “training set” of documents to teach the classifier to distinguish between responsive and non-responsive documents. Supervised learning methods are generally more static, while active learning methods involve more interaction with the tool and more iteration. Knowledge engineering approaches (a.k.a. “rule-based” methods) involve the construction of linguistic and other models that replicate the way that humans think about complex problems. Both approaches can be effective when properly employed and validated. At this time, only active machine learning and rule-based approaches have been shown to be effective for technology-assisted review. Service providers should be prepared to tell their clients what is “under the hood.”
What is the number one mistake practitioners should avoid when using these tools?
Not employing proper validation protocols, which are essential to a defensible process. There is widespread misunderstanding of statistics and what they can and cannot tell us. For example, many service providers report that their tools achieve 99% accuracy. Accuracy is the fraction of documents that are correctly coded by a search or review effort. While accuracy is commonly advanced as evidence of an effective search or review effort, it can be misleading because it is heavily influenced by prevalence, or the number of responsive documents in the collection. Consider, for example, a document collection containing one million documents, of which ten thousand (or 1%) are relevant. A search or review effort that identified 100% of the documents as non-relevant, and therefore, found none of the relevant documents, would have 99% accuracy, belying the failure of that search or review effort to identify a single relevant document.
What do you see as the key issues that will confront practitioners who wish to use predictive coding in the near-term?
There are several issues that will be played out in the courts and in practice over the next few years. They include: (1) How does one know if the proposed TAR tool will work (or did work) as advertised?; (2) Must seed or training sets be disclosed, and why?; (3) Must documents coded as non-relevant be disclosed, and why?; (4) Should TAR be held to a higher standard of validation than manual review?; and (5) What cost and effort is justified for the purposes of validation? How does one ensure that the cost of validation does not obliterate the savings achieved by using TAR?
What have you been up to lately?
In an effort to bring order to chaos by introducing a common framework and set of definitions for use by the bar, bench, and vendor community, Gordon V. Cormack and I recently prepared a glossary on technology-assisted review that is available for free download at: http://cormack.uwaterloo.ca/targlossary. We hope that your readers will send us their comments on our definitions and additional terms for inclusion in the next version of the glossary.
Maura R. Grossman, counsel at Wachtell, Lipton, Rosen & Katz, is a well-known e-discovery lawyer and recognized expert in technology-assisted review. Her work was cited in the landmark 2012 case, Da Silva Moore v. Publicis Group (S.D.N.Y. 2012).
