this post was submitted on 04 Sep 2024
595 points (94.5% liked)

Technology

59589 readers
3039 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 1 year ago
MODERATORS
 

Artificial intelligence is worse than humans in every way at summarising documents and might actually create additional work for people, a government trial of the technology has found.

Amazon conducted the test earlier this year for Australia’s corporate regulator the Securities and Investments Commission (ASIC) using submissions made to an inquiry. The outcome of the trial was revealed in an answer to a questions on notice at the Senate select committee on adopting artificial intelligence.

The test involved testing generative AI models before selecting one to ingest five submissions from a parliamentary inquiry into audit and consultancy firms. The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions with a focus on ASIC mentions, recommendations, references to more regulation, and to include the page references and context.

Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts. Then, a group of reviewers blindly assessed the summaries produced by both humans and AI for coherency, length, ASIC references, regulation references and for identifying recommendations. They were unaware that this exercise involved AI at all.

These reviewers overwhelmingly found that the human summaries beat out their AI competitors on every criteria and on every submission, scoring an 81% on an internal rubric compared with the machine’s 47%.

you are viewing a single comment's thread
view the rest of the comments
[–] ContrarianTrail@lemm.ee -2 points 2 months ago (16 children)

The AI we have today is the worst it'll ever be. I can only think of two possible scenarios where AI doesn't eventually surpass human on every single cognitive task:

  1. There's something fundamentally different about computer made of meat (our brains) that cannot be replicated in silica. I personally don't see this as very likely since both are made of matter and matter obeys the laws of physics.

  2. We destroy ourselves before we reach AGI.

Otherwise we'll keep improving our technology and inching forward. It may take 5 years or 50 but it wont stop unless either of the scenarios stated above is true.

[–] Ultraviolet@lemmy.world 7 points 2 months ago (5 children)

LLMs are fundamentally a dead end though. If we ever create AGI, it will be a qualitatively different thing from an LLM.

[–] ContrarianTrail@lemm.ee 7 points 2 months ago (4 children)

It's not obvious to me as to why this is for 100% certainty going to be the case. Even if it's likely true, there's still a chance it might not be.

load more comments (3 replies)
load more comments (13 replies)