Of course not. Here’s how leading AI labs mislead consumers, journalists, and each other.

Every month or two, an AI lab releases a new model that they claim leaves most, if not all, of their competitors in the dust. Recent examples include OpenAI’s GPT-4o, Anthropic’s Claude 3 Opus, and Google DeepMind’s Gemini Ultra. These announcements are generally followed by a large amount of hype from the tech media, like Ars Technica’s breathless “The AI wars heat up with Claude 3, claimed to have ‘near-human’ abilities.” People tend to take these claims seriously. They use these numbers to decide which labs are “ahead” of the others and often to decide which LLMs to use. However, they would do better to treat these reports with a healthy dose of skepticism.

Je teste depuis deux jours et ça marche bien, j'aime beaucoup l'interface pour l'instant !

cross-posted from: https://lemmy.world/post/18159531

An Apps Experiment

Introduction

This is an experiment I performed out of curiosity, and I have a few big disclaimers at the bottom. Basically, I've seen a lot of comments recently about one app or another not displaying something right. Lemmy has been around for a while now and can no longer be considered an experimental platform.

Lemmy and the apps that people use to access the platform have become an important part of people’s lives. Whether you are checking the app weekly or daily, and whether you use it to stay up on the news or to stay connected to your hobby, it’s important that it works. I hope that this helps people to see the extent of the challenge, and encourages developers to improve their apps, too.

How I did it

I wanted to investigate objectively how accurately each app displays text of posts and comments using the standard Lemmy markdown. Markdown is a standard part of the Lemmy platform, but not all apps handle it the same. It is basically what gives text useful formatting.

I used the latest release of each app, but did not include pre-releases. I only included apps that have released an update in the last 6 months, which should include most apps in active development. I was unable to test iOS-exclusive apps, so they are not included either. In all, 16 apps met the inclusion criteria.

Each app was rated in 5 categories: Text, Format, Spoilers, Links, and Images. I chose these mostly based on the wonderful Markdown Guide from @marvin@sffa.community, which was posted about a year ago in !meta@sffa.community (here).

I checked whether each app correctly displayed each category, then took the overall average. Each category was weighted equally. Text includes italic, bold, strong, strikethrough, superscript, and subscript. Format includes block quotes, lists, code (block and inline), tables, and dividers. Spoilers includes display of hidden, expandable spoilers. Links includes external links, username links, and community links. Images included embedded images, image references, and inline images.

In each case, I checked whether the display was correct based on the rules for Lemmy Markdown, and consistent with the author’s intent. In cases where the app recognized the tag correctly but did not display it accurately, that was treated as a fail.

Results

Out of a possible perfect 10, only 3 apps displayed all markdown correctly:

Jerboa (Official) - 10.0

Alexandrite - 10.0

Voyager - 10.0

Summit - 9.7

Interstellar - 9.3

Photon - 9.3

Lemmuy-UI - 9.3

Tesseract - 9.0

Thunder - 8.9

mlmym - 8.3

Quiblr - 7.9

Boost - 7.3

Sync - 6.2

Lemmynade - 6.2

Connect - 5.2

Racoon - 4.1

Disclaimers

Disclaimers

I Love Lemmy Apps (and their devs)

Lemmy apps devs work very hard, and invest a lot in the platform. Lemmy is better because they are doing the work that they do. Like, a LOT better. Everyone who uses the platform has access it through one app or another. Apps are the face of the entire platform. Whether an app is a FOSS passion project, underwritten by a grant, or generating income through sales or ads, no one is getting rich by making their app. It is for the benefit of the community.

This is not meant to be a rating of the quality or functionality of any app. An app may have a high rating here but be missing other features that users want, or users may love an app that has a lower rating. This is just about how well apps handle markdown.

This is pretty unscientific

You’ll see my methodology above. I’m not a scientist. There is probably a much better way to do this, and I probably have biases in terms of how I went about it. I think it’s interesting and probably has some valuable information. If you think it’s interesting, let me know. If you think of a better way, PM me and I’d be happy to share what I have so you don’t have to start from scratch.

My only goal is to help the community

I do think that accurately displaying markdown should be a standard expectation of a finished app. I hope that devs use this as an opportunity to shore up the areas that are lagging, and that they have a set of standards to aim for.

I don’t have any Apple things

Sorry. This is just Android and Web review. If someone would like to see how iOS apps are doing, please reach out and I’ll share how we can work together to include them.

From !canvas@toast.ooo

turns out I did my math wrong, so it’s a little less than 30 days

July 12th, 2024 @ midnight EDT

https://canvas.fediverse.events

✨ this year’s event also supports the entire fediverse not just Lemmy!

(you have to be able to make/receive text posts, like mastodon, lemmy, pixelfed, etc) (peertube accounts will not work)

you can get update announcements on other fedi platforms with @canvas@fediverse.events link

chat about Canvas on Matrix or Discord (they’re bridged)

C'est un même d'Amiral Poitou, toujours au sommet de l'esprit rance 🫡

inspiré par https://jlai.lu/post/7662480 j'ai passé les paroles sur le site Udio

I downloaded about 40 demos, will report back once I played all of them !