this post was submitted on 23 Nov 2024
556 points (95.9% liked)
Technology
62936 readers
3567 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each other!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
- Accounts 7 days and younger will have their posts automatically removed.
Approved Bots
founded 2 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
140Wh seems off.
It's possible to run an LLM on a moderately-powered gaming PC (even a Steam Deck).
Those consume power in the range of a few hundred watts and they can generate replies in a seconds, or maybe a minute or so. Power use throttles down when not actually working.
That means a home pc could generate dozens of email-sized texts an hour using a few hundred watt-hours.
I think that the article is missing some factor, such as how many parallel users the racks they're discussing can support.
Datacenter LLM tranches are 7-8 H100s per user at full load which is around 4 kW.
Multiply that by generation time and you get your energy used. Say it takes 62 seconds to write an essay (a highly conservative figure).
That's 68.8 Wh, so you're right.
Source: I'm an AI enthusiast
kW is a unit of instantaneous power; kW/s makes no sense. Note how multiplying that by seconds would cancel time out and return you power again instead of energy. You got there in the end, though.
Woop, noted, thanks