The NSW Supreme Court has introduced new rules on the use of generative artificial intelligence in legal proceedings. Its commencement coincides with a lawyer’s referral to a disciplinary regulator for submitting non-existent citations generated by ChatGPT.
On 3 February 2025, the NSW Supreme Court’s Practice Note regarding the use of generative artificial intelligence (AI) came into effect (Practice Note). The Practice Note contains essential guidelines for legal practitioners, offering new guidance for integrating generative AI tools such as ChatGPT into their legal practice.
Key guidelines for legal practitioners
The Practice Note reiterates and imposes:
The need for constant consideration of AI limitations and risks
- Legal practitioners must be aware of the limits, risks and shortcomings of the generative AI program used, including the potential for inaccuracies, otherwise known as “AI hallucinations”, and biases.
- Legal practitioners should be mindful that data entered into generative AI programs may be used to train their large language model, potentially making confidential information available to others with consequences for, among other things, legal professional privilege.
Prohibited use and disclosure
- Certain categories of documents and information must not be entered into a generative AI program unless the legal practitioner is satisfied that the information will remain within the controlled environment of the technology provider such that it is not made public, is used only in connection with the proceeding, and is not used to train the generative AI or any large language model. Relevant documents include, by way of two examples only, those subject to the implied undertaking and material produced on subpoena.
- Generative AI must not be used to generate the content of affidavits, witness statements, character references or other material intended to reflect the deponent or witness’ evidence and/or opinion, or other material tendered in evidence or used in cross examination. This is to avoid, for example, embellishment or diluting a witness’ evidence.
- The use of generative AI to draft or prepare the content of an expert report (or any part of an expert report) is prohibited without prior leave of the court. The Practice Note specifies when leave may be granted.
Verification
- Legal practitioners using generative AI programs must verify that every legal citation, academic authority, case law and legislative reference exists and is both accurate and relevant to the proceeding. Verification cannot be completed by using a generative AI program.
Permitted use
- The Practice Note recognises the potential benefits of generative AI. Therefore, subject to the prohibitions, it permits the use of generative AI programs for tasks such as generating chronologies, indexes, and witness lists, preparing briefs or drafting Crown Case Statements, and summarising or reviewing documents and transcripts. It also acknowledges that the Practice Note will be periodically reviewed due to the rapidly developing nature of generative AI.
Timely relevance of Practice Note
The commencement of the Practice Note arrives at a crucial time, following the recent case of Valu v Minister for Immigration and Multicultural Affairs (No 2) whereby a lawyer submitted court documents containing AI-generated, non-existent case citations. Judge Skaros referred the lawyer to the Office of the NSW Legal Services Commissioner (OLSC) after the lawyer admitted to filing an amended application and outline of submissions that '…contained citations to cases and alleged quotes…which were non-existent' and drafted with the assistant of ChatGPT.
The lawyer indicated that they used AI to identify relevant cases, but it provided non-existent case law. When the AI drafted a summary of cases, the lawyer believed it ‘read well’ and filed it with the court without checking its accuracy. This led to the court spending substantial time trying to locate the authorities. While the lawyer brought the issues to the court’s attention, apologised and was embarrassed, the court considered it appropriate for the referral to be made. In doing so, the Judge stated that the use of generative AI in legal proceedings is a live and evolving issue, and it was in the public interest of the OLSC to be made aware of such conduct.
Recommendations
The most recent example of AI misuse in a court proceeding reiterates the care and consideration that needs to be taken when using AI in legal practice. Whilst the Practice Note relates to its specified jurisdiction only, it is strongly suggested that the matters set out within it be adopted by practitioners in all jurisdictions and, of course, subject to any specific requirements in that jurisdiction. Doing so is likely to limit adverse consequences of AI use in legal practice.
View our recent video sharing insights on the Practice Note here.
Practice Note SC Gen 23
Valu v Minister for Immigration and Multicultural Affairs (No 2) [2025] FedCFamC2G 95 (31 January 2025)
