AI Hallucinations and NYS Legal Ethics: A Paralegal Memo
This memo, prepared by an IT paralegal, examines artificial intelligence and generative AI in the context of legal practice, with particular focus on New York State Rules of Professional Conduct. It defines key AI concepts, surveys landmark cases where AI-generated hallucinations led to court sanctions, reviews recent executive orders and NYS legislative proposals on AI-generated evidence, and analyzes how expert witness testimony has been affected. The memo concludes with practical recommendations — including verification protocols, attorney training, and ethical AI policies — to help law firms comply with professional standards while responsibly integrating AI tools.
- Introduction and Scope: Memo purpose and topics to be addressed
- Artificial Intelligence: Definitions and Legal Applications: AI types, key terms, and legal use cases
- Generative AI and Emerging Technology in Court Proceedings: Gen AI definition and VR in courtrooms
- Executive Orders, NYS Legislation, and Judicial Standing Orders: 2025 executive order, NYS bill, and disclosure requirements
- Case Law on AI Hallucinations: Sanctioned attorneys and fabricated AI citations
- Expert Witnesses and AI-Generated Evidence: AI hallucinations affecting expert testimony credibility
- NYS Rules of Professional Conduct and Practice Recommendations: Ethics rules, verification protocols, and firm policy guidance
✍️ How to write this paper — guide, tools & examples ▾
What makes this paper effective
- The memo format is well-matched to its professional audience — a supervising attorney — giving it practical clarity and a direct, action-oriented voice throughout.
- It integrates multiple types of authority (case law, executive orders, state legislation, and professional conduct rules) rather than relying on a single source type, lending credibility to each recommendation.
- The concluding recommendations map directly back to specific cases and rules discussed in the body, making the guidance both concrete and legally grounded.
Key academic technique demonstrated
The memo demonstrates effective use of case-law synthesis: rather than summarizing each case in isolation, the author groups Mata v. Avianca, Ex parte Lee, and Al-Hamin v. Star Hearthstone to establish a pattern of judicial response to AI hallucinations. This comparative approach elevates the analysis from description to legal argument, showing how a developing body of precedent collectively shapes attorney obligations.
Structure breakdown
The memo follows a traditional legal memorandum structure: a statement of assignment frames the scope, lettered discussion sections move from foundational definitions (AI, Gen AI) through regulatory context (executive orders, NYS bills) to applied analysis (case law, expert witnesses, professional rules), and a conclusion converts findings into actionable policy recommendations. This progression from concept to consequence to remedy is a model for professional legal writing.
Introduction and Scope
This memo was prepared in response to a request to review materials regarding Artificial Intelligence (AI), Generative AI (Gen AI), AI hallucinations, and the New York State (NYS) Rules of Professional and Ethical Conduct. It discusses these topics by providing definitions, case law analysis, legislative developments, and recommendations for legal practice.
Artificial Intelligence: Definitions and Legal Applications
The term Artificial Intelligence (AI) was first coined by John McCarthy, a leading figure in the field, in 1956 (Stanford's John McCarthy, 2011). AI refers to the capability of machines to mimic human cognitive functions such as learning, reasoning, and problem-solving (IBM, 2025). According to Marr (2018), AI is categorized into three types:
1. Strong AI — Systems designed to think like humans.
2. Weak AI — Systems that function efficiently without human-like reasoning.
3. AI as a Model — Using human reasoning as a framework without replicating human cognition.
In the legal profession, AI is used for contract analysis, legal research, and predictive analytics (Harvard Journal of Law & Technology, 2018). Key AI-related terms include:
1. Machine Learning (ML): Algorithms that improve with data exposure.
2. Large Language Models (LLMs): AI models that process and generate human-like text.
3. Neural Networks (NN): Computing systems modeled after the human brain for pattern recognition.
AI offers benefits such as efficiency, automation, and data-driven decision-making. However, risks include biases in algorithms, ethical concerns, and AI-generated misinformation (IBM, 2025).
Generative AI and Emerging Technology in Court Proceedings
Generative AI (Gen AI) is a subset of AI designed to generate new content — such as text, images, and music — based on training data (Zewe, MIT News). According to Stryker and Scapicchio (2024), Gen AI uses deep learning models to create outputs that resemble human-generated content. It is widely used in legal research, content creation, and even courtroom simulations.
Emerging technologies such as Virtual Reality (VR) are also influencing court proceedings. In a recent case, a Florida judge used a VR headset to visualize a crime scene, highlighting how technology can aid judicial decision-making (O'Driscoll, 2025).
Case Law on AI Hallucinations
AI hallucinations refer to instances where AI systems generate false or misleading information (Stanford HAI, n.d.). Hallucinations are especially problematic in the legal field, where factual accuracy is crucial.
In this case, attorneys cited fabricated case law generated by ChatGPT. The court sanctioned the attorneys for failing to verify the sources (Mata v. Avianca Inc., 2023).
The court flagged a brief containing non-existent precedents, possibly AI-generated, raising concerns about AI-assisted legal writing (Volokh et al., 2023).
Similar to Mata, this case involved attorneys using AI-generated case citations that proved to be hallucinations (Cohen, 2024).
Together, these cases demonstrate the serious risks of unverified AI use in legal practice.
References
Harvard Journal of Law & Technology. (2018, January 3). A primer on using artificial intelligence in the legal profession. https://jolt.law.harvard.edu/digest/a-primer-on-using-artificial-intelligence-in-the-legal-profession
Zewe, A. (n.d.). Explained: Generative AI. MIT News. https://news.mit.edu/2023/explained-generative-ai-1109
Grammarly. (2024, June 27). AI hallucinations: What they are and why they happen. https://www.grammarly.com/blog/what-are-ai-hallucinations/
Stanford HAI. (n.d.). AI on trial: Legal models hallucinate in 1 out of 6 (or more) benchmarking queries. https://hai.stanford.edu/news/ai-trial-legal-models-hallucinate-1-out-6-or-more-benchmarking-queries
Bill No. A08110 or S08390 ("Sets rules and procedures for the admissibility of evidence created or processed by artificial intelligence"). https://assembly.state.ny.us/leg/?default_fld=&bn=A08110&term=2023&Summary=Y&Actions=Y&Text=Y
Cohen, G. J. (2024, February 28). Court orders lawyer to pay opposing counsel after citing fake AI-generated cases. Legaltech News.
IBM. (2025, February 14). What is artificial intelligence (AI)? https://www.ibm.com/think/topics/artificial-intelligence
Judicature. (2024, February 7). Is disclosure and certification of the use of Generative AI really necessary? https://judicature.duke.edu/articles/is-disclosure-and-certification-of-the-use-of-generative-ai-really-necessary
Marr, B. (2018, February 14). The key definitions of artificial intelligence (AI) that explain its importance. Forbes. https://www.forbes.com/sites/bernardmarr/2018/02/14/the-key-definitions-of-artificial-intelligence-ai-that-explain-its-importance
New York State Bar Association. (2024, February 26). Professional standards. https://nysba.org/attorney-resources/professional-standards/
O'Driscoll, S. (2025, January 3). Florida judge wears VR headset to step inside simulation of crime scene. Newsweek.
Stanford University School of Engineering. (2011, October 25). Stanford's John McCarthy, seminal figure of artificial intelligence, is dead at 84. https://engineering.stanford.edu/news/stanfords-john-mccarthy-seminal-figure-artificial-intelligence-dead-84
Stryker, C., & Scapicchio, M. (2024, August 20). What is generative AI? IBM. https://www.ibm.com/topics/generative-ai
The United States Government. (2025, January 23). Removing barriers to American leadership in artificial intelligence. The White House. https://www.whitehouse.gov/presidential-actions/2025/01/removing-barriers-to-american-leadership-in-artificial-intelligence/
Volokh, E., et al. (2023, July 27). Court notes apparently nonexistent precedents cited in brief, suggests brief may have been partly AI-written. Reason. https://reason.com/volokh/2023/07/27/court-notes-apparently-nonexistent-precedents-cited-in-brief-suggests-brief-may-have-been-partly-ai-written
Al-Hamin v. Star Hearthstone, LLC, 2024COA128 (Colo. 2024).
Ex parte Lee, 673 S.W.3d 755, 756 (Tex. App.—Waco Jul. 19, 2023).
Kohls v. Ellison, Court File No. 24-cv-03754 (D. Minn.).
Mata v. Avianca Inc., Case No. 22-CV-1461 (S.D.N.Y. filed Feb. 22, 2022).
Mata v. Avianca, Inc., 2023 WL 3696209, 2023 U.S. Dist. Lexis 94323 (S.D.N.Y., May 4, 2023, order).
Matter of Weber, 2024 NY Slip Op 24258 (Surrogate's Court, Saratoga County Oct. 10, 2024).
Always verify citation format against your institution’s current style guide requirements.