Here's my overall concern with every legal benchmark in common law systems: it depends �

Here's my overall concern with every legal benchmark in common law systems: it depends 😇

ICYMI, Mercor released Apex Agents benchmarks and discovered that lawyers cannot easily be replaced by AI 👉🏽 https://lnkd.in/ee-KQZJg

So I was curious to see which legal task were so difficult and why AI would struggle.

Turns out, besides access to precise legal data, the answer also depends on several key legal aspects such as time or jurisdiction.

The 'golden' answers do not reflect such nuance so have a high likelihood to be bias towards a certain viewpoint.

As said to a good friend lately: we're subjective

If every legal answer was clearcut, we would never have court cases and wouldn't need lawyers 😇