Framework preview

Lawyer AI Competence

Measures whether lawyers can verify, supervise, and govern AI-assisted legal work with role-specific rigor.

Deployment Competence

Part 1: Technical Literacy

Can you explain what "tokens" are and why they matter for AI input/output?

confidence

Weight 1

important

Part 1: Technical Literacy

Do you understand the difference between a foundation model and a fine-tuned model?

confidence

Weight 1

important

Part 1: Technical Literacy

Can you describe what RAG (Retrieval Augmented Generation) is and why legal AI tools use it?

confidence

Weight 1

important

Part 1: Technical Literacy

Do you know what "temperature" settings control in AI output?

confidence

Weight 1

important

Part 1: Technical Literacy

Can you explain why AI models have knowledge cutoff dates?

confidence

Weight 1

important

Part 1: Technical Literacy

Can you define "hallucination" in the AI context and explain why it occurs?

confidence

Weight 1

important

Part 1: Technical Literacy

Do you understand "sycophancy" and how it manifests in legal research?

confidence

Weight 1

important

Part 1: Technical Literacy

Can you identify situations where AI is likely to produce unreliable output?

confidence

Weight 1

important

Part 1: Technical Literacy

Do you know what "misgrounding" means (correct statement, wrong citation)?

confidence

Weight 3

critical

Part 1: Technical Literacy

Can you explain why AI might perform differently on novel vs. common legal questions?

confidence

Weight 1

important

Part 1: Technical Literacy

Can you categorize legal tasks by AI risk level (low/medium/high)?

confidence

Weight 1

important

Part 1: Technical Literacy

Do you understand which matters should not use AI assistance?

confidence

Weight 1

important

Part 1: Technical Literacy

Can you identify when confidentiality concerns prohibit AI use?

confidence

Weight 3

critical

Part 1: Technical Literacy

Do you know how to evaluate vendor data handling practices?

confidence

Weight 1

important

Part 4: Ethical Compliance

Have you reviewed AI guidance from your primary state bar?

confidence

Weight 1

important

Part 4: Ethical Compliance

Do you know whether your jurisdiction requires AI disclosure in filings?

confidence

Weight 1

important

Part 4: Ethical Compliance

Are you aware of CLE requirements for AI competence?

confidence

Weight 1

important

Part 4: Ethical Compliance

Do you understand multi-jurisdictional compliance obligations?

confidence

Weight 1

important

Part 4: Ethical Compliance

Do you know how your AI tools handle client data?

confidence

Weight 1

important

Part 4: Ethical Compliance

Have you reviewed vendor agreements for data protection?

confidence

Weight 1

important

Part 4: Ethical Compliance

Do you avoid entering confidential information into non-secure tools?

confidence

Weight 3

critical

Part 4: Ethical Compliance

Do you understand where AI processing occurs geographically?

confidence

Weight 1

important

Part 4: Ethical Compliance

Do you treat AI vendors as supervised service providers?

confidence

Weight 1

important

Part 4: Ethical Compliance

Have you established firm policies for AI use?

confidence

Weight 1

important

Part 4: Ethical Compliance

Do you supervise staff AI use appropriately?

confidence

Weight 1

important

Part 4: Ethical Compliance

Do you review AI-assisted work before submission?

confidence

Weight 1

important

Training & Development

Part 5: Ongoing Education

Do you track state bar AI guidance updates?

boolean

Weight 1

nice to have

Part 5: Ongoing Education

Do you follow legal AI developments and research?

boolean

Weight 1

nice to have

Part 5: Ongoing Education

Are you aware of the Stanford hallucination research findings?

boolean

Weight 1

important

Part 5: Ongoing Education

Do you monitor AI-related ethics opinions and court decisions?

boolean

Weight 1

important

Part 5: Ongoing Education

Have you completed AI-focused CLE programs?

boolean

Weight 1

important

Part 5: Ongoing Education

Have you taken courses on legal technology or AI?

boolean

Weight 1

important

Part 5: Ongoing Education

Do you participate in AI-focused professional development?

boolean

Weight 1

important

Part 5: Ongoing Education

Have you obtained any AI competency certifications?

boolean

Weight 1

important

Part 5: Ongoing Education

Do you regularly use AI tools in appropriate contexts?

boolean

Weight 1

important

Part 5: Ongoing Education

Do you learn from verification failures when they occur?

boolean

Weight 1

nice to have

Part 5: Ongoing Education

Do you share learnings with colleagues?

boolean

Weight 1

nice to have

Part 5: Ongoing Education

Do you contribute to firm AI policies and training?

boolean

Weight 1

important

Technical Infrastructure

Part 2: Verification Skills

Do you maintain records of AI tools used and prompts provided?

evidence

Weight 1

important

Part 3: Prompt Engineering

Do you structure prompts with clear, specific instructions?

confidence

Weight 1

important

Part 3: Prompt Engineering

Do you provide relevant context to guide AI responses?

confidence

Weight 1

important

Part 3: Prompt Engineering

Do you know how to request citations and supporting authority?

confidence

Weight 3

critical

Part 3: Prompt Engineering

Can you prompt for acknowledgment of uncertainty?

confidence

Weight 1

important

Part 3: Prompt Engineering

Do you understand how to avoid leading the AI toward desired conclusions?

confidence

Weight 1

important

Part 3: Prompt Engineering

Do you ask follow-up questions to test AI reasoning?

confidence

Weight 1

important

Part 3: Prompt Engineering

Do you prompt for contrary authority or counterarguments?

confidence

Weight 1

important

Part 3: Prompt Engineering

Do you test AI responses against known edge cases?

confidence

Weight 1

important

Part 3: Prompt Engineering

Do you verify AI does not simply agree with incorrect premises?

confidence

Weight 1

important

Evidence & Documentation

Part 2: Verification Skills

Do you independently verify every case citation in AI output?

evidence

Weight 3

critical

Part 2: Verification Skills

Do you confirm that cited cases actually support stated propositions?

evidence

Weight 2

important

Part 2: Verification Skills

Do you check that holdings are accurately characterized?

evidence

Weight 3

critical

Part 2: Verification Skills

Do you verify quotations against original sources?

evidence

Weight 2

important

Part 2: Verification Skills

Do you confirm cases have not been overruled or distinguished?

evidence

Weight 3

critical

Part 2: Verification Skills

Can you identify logical gaps in AI-generated analysis?

evidence

Weight 2

important

Part 2: Verification Skills

Do you check AI output against your own understanding of the law?

evidence

Weight 2

important

Part 2: Verification Skills

Can you recognize when AI has missed relevant contrary authority?

evidence

Weight 2

important

Part 2: Verification Skills

Do you verify that AI correctly applies law to your specific facts?

evidence

Weight 2

important

Part 2: Verification Skills

Can you identify when AI has oversimplified nuanced legal questions?

evidence

Weight 2

important

Part 2: Verification Skills

Do you document verification steps for AI-assisted work?

evidence

Weight 2

important

Part 2: Verification Skills

Can you demonstrate the verification process if challenged?

evidence

Weight 2

important