This comprehensive analysis examines the intersection of artificial intelligence and legal practice, focusing on landmark cases involving AI technology and their outcomes. The essay explores AI hallucinations in legal contexts, professional ethical conduct rules, and emerging legislative frameworks governing AI use in the judicial system. Through case law analysis and regulatory review, it demonstrates the critical need for updated professional standards in an AI-driven legal landscape.
MEMO
Re: Artificial Intelligence, Generative AI, Hallucinations, and NYS Rules of Professional and Ethical Conduct
You have asked me to review materials regarding Artificial Intelligence (AI), Generative AI (Gen AI), AI hallucinations, and the New York State (NYS) Rules of Professional and Ethical Conduct. This memo discusses these topics, providing definitions, case law analysis, legislative developments, and recommendations for legal practice.
The term Artificial Intelligence (AI) was first coined by John McCarthy, a leading figure in the field, in 1956 (Stanford’s John McCarthy, 2011). AI refers to the capability of machines to mimic human cognitive functions, such as learning, reasoning, and problem-solving (What is Artificial Intelligence (AI)?. IBM). According to Marr (2018), AI is categorized into three types:
1. Strong AI – Systems designed to think like humans.
2. Weak AI – Systems that function efficiently without human-like reasoning.
3. AI as a Model – Using human reasoning as a framework without replicating human cognition.
In the legal profession, AI is used for contract analysis, legal research, and predictive analytics (A primer on using artificial intelligence in the legal profession, 2018). AI-related terms include:
1. Machine Learning (ML): Algorithms that improve with data exposure.
2. Large Language Models (LLMs): AI models that process and generate human-like text.
3. Neural Networks (NN): Computing systems modeled after the human brain for pattern recognition.
AI offers benefits such as efficiency, automation, and data-driven decision-making. However, risks include biases in algorithms, ethical concerns, and AI-generated misinformation (What is Artificial Intelligence (AI)?. IBM).
B. Generative Artificial Intelligence (Gen AI) Defined and Explained & Emerging Tech in Court Trials
Generative AI (Gen AI) is a subset of AI designed to generate new content, such as text, images, and music, based on training data (Adam Zewe | MIT News). According to Stryker and Scapicchio (2024), Gen AI uses deep learning models to create outputs that resemble human-generated content. It is widely used in legal research, content creation, and even courtroom simulations.
Emerging technologies, such as Virtual Reality (VR), are also influencing court proceedings. In a recent case, a Florida judge used a VR headset to visualize a crime scene, highlighting how technology aids judicial decision-making (O’Driscoll, 2025).
On January 23, 2025, the U.S. government issued an Executive Order on AI aimed at removing barriers to AI leadership (The United States Government, 2025). The order seeks to promote AI innovation while addressing ethical concerns, establish regulatory frameworks for AI implementation, ensure AI’s safe and beneficial use across industries.
In NYS, Bill No. A08110 or S08390 proposes rules on the admissibility of AI-generated evidence, focusing on authenticity and transparency (Bill No. A08110 or S08390). These legal measures reflect increasing awareness of AI\\\\\\\'s impact on the judicial system.
AI hallucinations refer to instances where AI systems generate false or misleading information (AI hallucinations: What they are and why they happen, 2024). Hallucinations are problematic in the legal field, where factual accuracy is crucial (AI on trial: Legal models hallucinate in 1 out of 6 (or more) benchmarking queries).
1. Mata v. Avianca Inc. (2023) – In this case, attorneys cited fabricated case law generated by ChatGPT. The court sanctioned the attorneys for failing to verify sources (Mata v. Avianca Inc (2023)).
2. Ex parte Lee (2023) – The court flagged a brief containing non-existent precedents, possibly AI-generated, raising concerns about AI-assisted legal writing (Volokh et al., 2023).
3. Al-Hamin v. Star Hearthstone, LLC (2024) – Similar to Mata, this case involved attorneys using AI-generated case citations that proved to be hallucinations (Cohen, 2024).
These cases demonstrate the risks of unverified AI use in legal practice.
Expert witness testimony is another area affected by AI-generated hallucinations.
In this NY Surrogate’s Court case, an expert relied on AI-generated data, raising concerns about the authenticity of evidence (Matter of Weber, 2024 NY Slip Op 24258).
This case involved an expert witness allegedly using AI-generated materials in a court declaration, casting doubt on the testimony’s credibility (Order Granting in Part and Denying in Part Plaintiffs’ Motion, 2025).
Courts are addressing AI-related issues by issuing Judicial Standing Orders requiring attorneys to disclose and certify AI use in legal filings.
A recent article questions whether such disclosure requirements are necessary or if they place undue burdens on legal professionals (Is disclosure and certification of the use of Generative AI really necessary? 2024). While these orders promote transparency, they may also limit the efficiency AI provides.
The NYS Rules of Professional Conduct require attorneys to maintain competency and confidentiality when using emerging technologies (Professional standards, 2024). Competency Rule (Rule 1.1) states that attorneys must understand the technology they use to provide competent representation. Confidentiality (Rule 1.6) states that AI tools must not compromise client data. Attorneys must balance AI’s advantages with ethical responsibilities, ensuring they verify AI-generated content before use.
Based on current case law, legal standards, and ethical considerations, law firms should adopt the following measures:
1. AI Verification Protocols: Implement mandatory AI-generated content verification to prevent misinformation in legal filings.
2. Training and Education: Require attorneys and paralegals to complete AI literacy training to understand AI capabilities and risks.
3. Ethical AI Use Policy: Establish firm-wide policies that align with NYS professional conduct rules, ensuring responsible AI usage.
4. Court Compliance: Adhere to judicial standing orders regarding AI use disclosure.
A primer on using artificial intelligence in the legal profession. Harvard Journal of Law &
Technology. (2018a, January 3). https://jolt.law.harvard.edu/digest/a-primer-on-using-artificial-intelligence-in-the-legal-profession
AI hallucinations: What they are and why they happen. Grammarly. (2024, June 27).
Ai on trial: Legal models hallucinate in 1 out of 6 (or more) benchmarking queries. Stanford
HAI. (n.d.). https://hai.stanford.edu/news/ai-trial-legal-models-hallucinate-1-out-6-or-more-benchmarking-queries
Bill No. A08110 or S08390 (“Sets rules and procedures for the admissibility of evidence created
or processed by artificial intelligence”),
https://assembly.state.ny.us/leg/?default_fld=&bn=A08110&term=2023&Summary=Y&Actions=Y&Text=Y&Committee%26nbspVotes=Y&Floor%26nbspVotes=Y#A08110
Cohen, G. J. (2024, February 28). Court orders lawyer to pay opposing counsel after citing fake
AI-generated cases. Legaltech News.
Generative AI: What is it, tools, models, applications and use cases. (n.d.-a).
IBM. (2025, February 14). What is Artificial Intelligence (AI)?. IBM.
You’re 80% through this paper. Sign up to read the full paper.
Sign Up Now — Instant Access Already a member? Log inAlways verify citation format against your institution’s current style guide requirements.