More commentary on what I think is an important topic that is being absolutely ignored by the “legitimate, mainstream news” media. First, the story, from this week’s issue.
You Don’t Say?
“Losing my voice has been a massive deal for me because it’s such a big part of who I am,” says Joyce Esser from her home in an unspecified part of the U.K. She has motor neuron disease (called ALS in the U.S.), and can only speak that sentence thanks to an A.I.-powered system that has learned to perfectly mimic her voice, trained by recordings made earlier. She also has difficulty swallowing, and probably only has a few years more to live, but the ability to speak helps her function in the world. When getting ready to go out with her husband, she had her system tell him, “Come on, Hunnie, get your arse in gear!” and adding, “I’d better get my knickers on too!” Uh oh. “The next day I got a warning from ElevenLabs that I was using inappropriate language and not to do it again,” she told a reporter. It was “normal British banter between a couple getting ready to go out.” But the next day, the U.S.-based company banned her from using their system anymore. “I’d just got my voice back and now they’d taken it away from me.” A reporter asked the company what the problem was, and was referred to their “prohibited use policy,” though that doesn’t list any prohibitions on “inappropriate language.” She complained, and they reinstated her. “People living with MND should be able to say whatever is on their mind, even swearing,” says Richard Cave of the U.K.’s MND Association. “There’s plenty to swear about.” (RC/MIT Technology Review) …Not creepy: telling a spouse to “get your arse in gear.” Creepy: a company listening in to what you say to your spouse in private.
First, a Note About My Source
As True is News Commentary, and by extension — particularly over time, as a body of work — social commentary, I have to set “rules” about the sources we use to ensure it stays in focus. The main rule is that sources are “legitimate, mainstream, news” sites, such as newspapers, TV news operations, and (less so as they’ve declined) news magazines.
MIT Technology Review — my source for the above story — doesn’t fit that rule. I’m pretty reluctant to violate my own rule, but I waited a week after spotting that story, and was shocked that no “legitimate, mainstream, news” site, at least that I could find, picked up on the story.
Yes, it was in the U.K.’s Daily Star tabloid, as well as Vice, Tech Times, and Boing Boing. None of those qualify in my eyes as “legitimate, mainstream, news” sites. At least the original source all of those sites used as their own source — MIT Technology Review, published by the Massachusetts Institute of Technology* with roots going back as a publication to 1899, is pretty legit, even if not “mainstream” in the “news” sector.
So I went with it because the story needs to be told widely, even though the legit/mainstream folks would prefer to write about dogs getting their heads stuck in a jar, to choose another story from this week that I saw covered by multiple news sites, even in Canada.
*Yeah, it’s really the Massachusetts Institute of Technology Technology Review. Shrug!
Side Rant
In the Olden Days, one bought a product and it belonged to us. Today we buy a product and someone wants to figure out how to charge a recurring fee for us to use that product.
Oh, you want an example? I wear an Oura “smart ring” that tracks my sleep and activity, but I have never recommended it because shortly after I got it, the company decided that selling a $299 bit of hardware, which needs to be replaced around every 3 years, wasn’t enough: if you want to actually review all the data that it gathers, you have to pay a $5.99/month “membership fee” to get it. Really.
Yes, I know that it costs money to keep up development, but see “$299 around every 3 years.” The company was at least honorable about adding the membership: it “grandfathered” (for life) those of us who bought in before they got that idea. You can’t get that, so that’s why I don’t recommend the ring. I was willing to at $100/year, but not at $184/year.
It’s why cell companies can “give” you an $800 smartphone: the $100/month to run it is awfully lucrative.
Yep, I sell subscriptions too. Each week you get a brand new product, created specifically for that week. You don’t buy that and pay $5.99/month to be able to open old issues to read them again.
Where Was I?
Oh yes, ElevenLabs, who created an amazing thing that really helps people who have lost their voices: it gives them their voices back — a piece of their personal identity. For many, their dignity, their ability to function in the world as their bodies slowly fail.
And these vile MFers think it’s OK to monitor what people say to each other in the privacy of their own homes, and lightly take that dignity away for daring to use their system to utter the world “arse” — in the U.K. no less! Talk about getting their knickers in a twist.
Their GOOHF cards are hereby VOID.
How dare they?! Joyce Esser, and everyone like her, can no longer fully enjoy the freedom they’ve regained because they now have to spend some amount of their remaining energy worrying that their own voices will be taken away from them on a whim by some stick-up-their-arse minimum-wage stooge listening in to what they say to their loved ones from 5,000 miles away.
It’s inhuman.
ElevenLabs must guarantee — in writing — that the people who are using this device have privacy, that they can say what they damned well want, in private or in public, using their own voices, and the company won’t monitor the content of such speech.
And they must guarantee — in writing — that they will never, ever, do anything to deprive those customers of their voices for any reason.
Or they need to get the hell out of the medical device business, preferably by spinning this device off to a Benefit Corporation run by people with actual ethics.
“People living with MND should be able to say whatever is on their mind, even swearing,” said Richard Cave of the U.K.’s MND Association, as quoted above. “There’s plenty to swear about.” Especially with these threats looming over their heads.
ElevenLabs was in a position to be absolute heroes. They have instead chosen to be villains. It’s not too late for them to turn things around. I hope they will.
Note Regarding Comments: I will be pretty open about comments, including allowing horror stories about corporations limiting the use of medical devices or charging ridiculous ongoing fees for medical devices. That is such wide and fertile ground that I won’t be approving such comments for other devices, even cell phones. Thanks for helping to keep this focused.
– – –
Bad link? Broken image? Other problem on this page? Use the Help button lower right, and thanks.
This page is an example of my style of “Thought-Provoking Entertainment”. This is True is an email newsletter that uses “weird news” as a vehicle to explore the human condition in an entertaining way. If that sounds good, click here to open a subscribe form.
To really support This is True, you’re invited to sign up for a subscription to the much-expanded “Premium” edition:
Q: Why would I want to pay more than the minimum rate?
A: To support the publication to help it thrive and stay online: this kind of support means less future need for price increases (and smaller increases when they do happen), which enables more people to upgrade. This option was requested by existing Premium subscribers.
What kind of pieces of $hit would come up with a rule on how someone must act, in the privacy of their own home, in order to be ALLOWED to use the software/hardware that they purchased? OMG!
What comes next? Those with artificial limb replacements will have them PULLED if they dare to have sex at the same time they are put together, as it were? “Well, Dan — I’m sorry, but we suspect that while you were courting this girl, you got to at LEAST third base in the front seat of your truck — and you were wearing our hand. We’re going to have to take it away….”
Sheesh.
—
I knew folks would quickly grasp the horror of this concept. -rc
That is *horrible* behavior from that company!
The nice thing about most things in life; if you don’t like what some company is doing, there are plenty of others out there that don’t. For me, it was Ring. I didn’t like the idea of paying a monthly subscription to have access to the videos I was recording with my Ring camera, so I bought a different camera that allows me to record to an SD Card. Same thing with your ring! I’m sure there’s another company out there that will monitor your vitals without making you pay a subscription.
But the problem with new emerging technology is there often isn’t any competition (yet). How many other companies can train an AI to mimic your voice and be used by people with Joyce’s condition? I imagine not many.
Companies like that often implement anti-consumer policies and get away with it for years before someone else eventually forces them to change for the better. For an example: look what happened to Verizon! In 2011 they tried to drop their “unlimited” plan and when T-Mobile stole huge amounts of their customers with a similar plan, Verizon was forced to cave and re-institute a version* of unlimited.
*It’s not fully unlimited like in 2011. They have a cap they can institute.
—
Indeed I’ve stayed with my ring — I’m around 3 years into my second one (“3rd generation” as I started with the 2nd) because they grandfathered the free membership for me (and Kit). Yes, there are competitors …and most of them charge a monthly fee. -rc
People are self-censoring all over social media now. Soon (especially among younger people I suspect) this will be normalized. People no longer “die” – they become “unalive”. You “k!ll” a malfunctioning program. Your car battery “d!es”. Letters are blurred out or blocked at the slightest hint of impropriety.
We all know what this is: it’s Newspeak. When you control the words, you control the mind. Long Live George Carlin!
—
I suspect no one who still has the ability to think will wonder why I felt it necessary to link “Newspeak” to an article about what it is. Sigh. -rc
1984 arrived 40 years late!
While I absolutely love Carlin, I think you mixed up your Georges. It’s Orwell you referenced!
—
And I decided just to let it go, and let someone else make the correction. I am not disappointed. 🙂 -rc
I don’t think she was referencing Orwell at all but was alluding to George Carlin’s comedy bit about 7 dirty words you can’t say on TV. Although both have relevance in this instance.
Possibly not, but “Newspeak” was Orwell’s concept first, in the 1940s. Carlin came along later, in the 1960s. Undoubtedly, Carlin read Orwell’s “1984” as part of his high school experience, as I did. Carlin’s 7 dirty words were part of his early 70s work.
As a techie, I’ll allow logging of data for testing and debugging purposes, and those purposes ONLY. “Service improvement” is another way of saying they want to use your private info to figure out something else to sell you. Regardless of them restoring the service to Joyce, imagine the chilling effect of knowing that these &^%#$ have been monitoring her communications in the past and only their promise says they aren’t monitoring it still. I hope that Joyce is of the mindset that if they are still listening they should expect to get their ears blistered.
—
I’m not sure where you got the impression that they promised they won’t continue to monitor. I just reviewed the MIT article, and it doesn’t say that. Indeed, it says that “There are rules against threatening child safety, engaging in illegal behavior, providing medical advice, impersonating others, interfering with elections, and more. But there’s nothing specifically about inappropriate language.” — so I have to assume they do monitor (though, of course, “terms of use state that the company does not have any obligation to screen, edit, or monitor content” [emphasis added] “but add that it may ‘terminate or suspend’ access to its services when content is ‘reasonably likely, in our sole determination, to violate applicable law or [the user] Terms.”, and that “ElevenLabs has a moderation tool that ‘screens content to ensure it aligns with our Terms of Service,’ says Dustin Blank, head of partnerships at the company.” That’s why I’m so strong in my editorializing that ALL of this needs to stop. I have edited my link to MIT to go directly to that source article, rather than the magazine’s home page. -rc
Replying to Randy’s comment, not Liam’s here.
It’s actually worse than even that. Go look up what the Third Party Doctrine is in US law, if you don’t already know.
For those who don’t: It’s basically case law in US courts that says that in most circumstances, the government doesn’t need a search warrant to seize records held by third parties. They don’t even need to notify you that they seized your records, and might even give whoever they seized them from a judicial gag order to prevent them from telling you.
The legal theory is that by sharing your data like that, it’s no longer a secret.
So everything any customer of ElevenLabs says where the system can hear it is searchable by the US government, and fully admissible in court, even if it would normally be privileged, such as that customer talking to their lawyer or discussing their medical care with their doctor!
—
Good point that I hadn’t thought of yet. -rc
I *totally* agree with the comment from Melody, Houston Texas. Doing that for an artificial limb would be terrible but imagine a *pacemaker* firm doing that!
Them: “We see your activities are straining our pacemaker. What you are doing is immaterial. The fact remains that your activities are straining our pacemaker. That voids the user agreement for our pacemaker. As of day&time we will be remotely disabling it. If this causes your death, too bad. You shouldn’t have been straining our pacemaker.”
I also thoroughly applaud your “Their GOOHF cards are hereby VOID” statement. Very nice touch.
[ I totally agree on the “monitoring and censoring private conversations” part of the article.]
I’d say that the big question around the appropriateness of charging a subscription is whether the company provides an ongoing service.
When you buy a smartphone, nobody charges you anything to use it. What they charge you for is to use their *service* of providing cellular connectivity. If you don’t want to connect to the cellular net, the ongoing cost is zero.
Yes, you can buy cameras that record locally and don’t charge a service fee. Don’t complain when they run out of storage and you can’t expand them, or when the burglar steals the camera and its SD card, or when you can’t watch the camera from the other side of town. All of those are features provided by the *service*.
I recently bought a Subaru. I can drive it as long as I want without paying Subaru another dime. But if I want to use their remote-start, tracking, emergency-response, remote-vehicle-status *service*, I have to keep paying for it. Something’s got to pay to keep those satellites flying.
Yes, some products include their service in the original purchase price. I suspect that most of those rely on repeat sales… or are doomed to fail, because those services are not free to provide.
And yes, some products (like your Oura ring) don’t really need a service. There’s probably nothing exciting that the service is doing; most things that it does could be done locally on your phone. And they have a recurring revenue stream: replacing the ring when the battery dies.
By all means look at the recurring costs of owning a product, but also look at the ongoing expenses the vendor has to keep them running. Unless everything associated with the device is local, there’s a cost to the vendor that has to be paid by the customer, somehow. TANSTAAFL.
—
I generally agree: it echoes my point about my newsletter subscriptions. At my Colorado home, I had security cameras. Everything was not only recorded, I could access the cameras and the recordings remotely. The only “service” needed was an Internet connection to the hardware in my home; that was the only monthly fee I paid. The recordings were kept on a 1TB hard drive that was secure: all the cameras could be stolen and I still had that device, similar to commercial security camera systems because …it was a commercial security camera system. Companies hire people who understand security to get systems that serve them. Most (not all) consumers get something that’s cheap without figuring the yearly cost. My “expensive” security camera system was a lot cheaper than cheap cameras if costs were looked at over the long run. -rc
I was using Google maps to help me navigate to a location through Houston, TX, rush hour traffic. It was verbally providing directions. At some point, I picked up my phone to look at the map closer, and the screen went to a page filled with text and multiple choices. I yelled the F expletive. The google voice broke in, calling me BY NAME, and said: “Harmon, that wasn’t very nice. Please use nicer language when using Google maps”.
Fuck that. Somebody was listening.
I think replies like these are a feature (some would call them “easter eggs”) built into the software of Google Assistant, likely intended to be funny. It doesn’t take a human listener to recognize an expletive.
Some time ago I was talking to my family in the kitchen, and suddenly my phone started to say something. It thought it had heard “Hey Google!” I said: “I wasn’t talking to you.” and it was like: “Oops!”
This is a REALLY common issue with Augmentative and Assistive Communication (AAC) devices — the censoring part, I mean, not the monitoring part. (That’s horrendous, too, but I think most people commenting before me have covered that pretty thoroughly!) I’m in a lot of online writing communities, and I know of a lot of people who have been really excited to finally get a speech-to-text program that allows them to write, or communicate online, effectively — it’s hard to find ones that work with certain accents, only to immediately discover that anything remotely considered “profane” gets censored. I’ve never heard of one actually revoking your access to the service for using it, but I’ve seen plenty of complaints from people that they say something with a profanity, even a mild one, and it gets changed to a line of asterisks or bowlderized into something (purportedly) sanitized and innocent.
It’s a problem I’ve noticed with captions, too. Whether automatically generated by the video site or put on by the creator, a lot of times people will say a profanity, slur, or other strong language in the audio, but censor it in the captions. Which is not helpful! If you’ve bleeped out the audio, censor the caption by all means, but closed captioning is supposed to be a faithful transcription of what is being said so that someone who is d/Deaf or hard of hearing can follow what is being said.
Disabled people are not infants. They don’t need to be coddled or protected from “bad language” or “adult activity”. They are just as capable of making those decisions as anyone else.
Also…I’m remembering the story you covered in an issue of True fifteen or twenty years ago about “clbuttic mistakes” (which I only remember because you were recording one story a week as a short video at the time, and I remember the outtake of you giggling madly). You start filtering for profanity, you’re going to start filtering completely innocuous words because they MIGHT have a hidden swear word in them.
Or, to paraphrase Tom Lehrer, “All words can be indecent words, though recent words are bolder / For filth, I’m glad to say, is in the mind of the beholder…”.
—
The clubuttic story and video are classic of True’s genre. -rc
SaaS (software as a service) is definitely the model companies love. Upkeep and updates paid for by customers? Groovy. But, the issue of ownership is tricky in those cases if it comes tied to a piece of hardware. I’d hope within a decade or so open source libraries (aka software anyone can code for free) would become standard for this kind of thing. Company dies or stops supporting a device, someone else can step in and keep it running.
In the case of ElevenLabs, I looked at their site, and they don’t seem to offer a tool designed for what Essie is doing. They seem to focus on API’s and such to embed their tech into other applications. Their market is mostly for voiceovers and similar applications.
I’m guessing that they have limits because they don’t want their tool used in “adult” applications. Think about a chat bot version of a phone sex service. Makes sense, as they’d be open to litigation if a minor used the service, even if they were just the tech used in the service and not the owners of the service.
Because of that, I’m thinking they didn’t have this “use case” in mind when developing the tech. It’s extremely common in software development. You get tunnel vision. You have a specific use for your tool in mind, but users are endlessly creative. My job for about 25 years was software QA. Most of that was finding bugs, but part of it was exactly this: thinking of ways users might use the software that would cause issues. If I worked for these guys, I would have brought up this use case specifically. People using it as their voice. Can’t censor that.
I’m willing to bet this example just flew right over their heads. They aren’t marketing it as a tool for people who have lost the ability to speak, they’re just marketing it as a general AI voice generator with a push towards selling it to third party devs to use in their projects.
To sum up: I doubt this has evil intent. It’s the wild west for this kind of tech, and developers have a hard time thinking about humans sometimes. I spent more than two decades explaining to them why “well, then they shouldn’t do that” isn’t an answer to a “usability bug” I’d reported.
—
Good analysis — and exactly why I suggested that they spin this out to a Benefit Corp. so that it can be developed specifically for this market, which would also sever the liabilities you point out. -rc
I’m a part time AAC (augmentative and alternative communication) user, which includes text to speech when that’s the best method. This story horrifies me. No one (or company) should have access to someone’s private conversation/speech without explicit consent. Not just because adults should be allowed to swear if they want (which they should, no matter how they communicate) but also because private speech should remain exactly that. (Arse isn’t even really considered swearing here, and is fine to use on TV or radio when kids are around, but that’s beside the point.)
Sometimes I have conversations about things I’ve signed non-disclosure agreements for. If I reveal what they’re about before they’re public, not only would I personally be in a lot of legal trouble, but people’s jobs would be at risk.
It’s easy for me to make sure those conversations stay private if using my mouth voice, encrypted messaging apps or AAC that remains entirely on my device, but not if using a system like the one in the story. I had a quick look at the website and it isn’t made clear that they’ll be monitoring everything you say using their service. And no, putting it in the terms of service isn’t clear enough.
Most current text to speech voices are either generic or cost a lot to personalise, which is out of reach for many disabled people. On first glance this looks like the perfect solution, and I can easily see people using it not knowing their conversations would be monitored. And the consequences for others could be much worse than for me.
It’s not just swearing or confidential topics either. They don’t allow using it to acquire alcohol, tobacco, or any drugs (including over the counter or prescribed) without their permission. So if you don’t ask them first, you can be banned for ordering a beer or asking someone to get you some aspirin. A quarter of the preset buttons on my text to speech app are medication related, because that’s something that’s important to communicate no matter how my speech is. And adults are allowed to smoke or drink if they want, even if they’re disabled.
I guess for now it’s safer to keep sounding like a robot (or paying a lot for a personalised voice). This is one of the genuinely good uses for AI which I’ve been excited for, but not like this.
It’s really sad that companies behave this way. It’s one thing to moderate ones speech, and I can understand why platforms, like Facebook, restrict what they allow, but for services like this, I can’t really understand what the company would benefit from doing things like this
Appalling behaviour, I’m surprised your friends at the Tech Enthusiast Hour haven’t weighed in yet as they use ElevenLabs.
There is now a purely local alternative as described in this The Register article.
It’s from a Palo Alto startup and can run on a “Linux box with a reasonably modern Nvidia graphics card with at least 8 GB of vRAM.”. Being Linux, no service fees, although some technical ability may be required to set it up initially.
Keep up the good work and enjoy your floating lifestyle.
—
Both Leo and Gary (the TEH hosts) subscribe to Premium, so they’re likely both aware of the issue. So maybe it’ll be covered in the next episode. -rc
There was something similar to ElevenLabs’ hijacking of a person’s use of the prosthesis in a story collection titled “Blue Champagne.” The main character’s full body prosthesis was free to use, but the contract granted exclusive 24/7 audio/video recording rights that the provider could monetize as they saw fit. Including any and all ‘private’ moments.
The ‘Third Party Doctrine’ I don’t much care for, either, nor the notion that my pacemaker might be seized.
There are quite frankly less stringent restrictions on what one can do with a firearm or a piece of gardening equipment for that matter which could bring about far more harm and probable harm and much less oversight.
Ridiculous.
Re: artificial limbs
What if what I want to use my new appendage for goes against someone else’s idea of what is morally or ethically acceptable?
Get your mind out of the gutter (though it could be that)!
Maybe I want to use it to turn the pages of a textbook on CRT?
I completely agree with the “no censoring” position. (“Monitoring” is a trickier question, because technologies like this likely involve processing the content at the vendor’s servers; is that “monitoring”? But if we said “no recording” and “no human monitoring” we’d probably cover the bases.)
However, there is a flip side. There have been a number of high-profile lawsuits in recent years claiming that some communications platform in some way contributed to violence, whether self-harm or against others. It needs to be made ABSOLUTELY CLEAR that the platform is NOT liable if you use it to hire a hit man, or if you use it to buy your suicide drugs, or if you use it to conspire to blow up a bus.
—
Strangely specific. I like people with slightly warped minds. -rc
When I searched for an article about this, I did find an MSN article.
—
Definitely always look carefully for sources! That’s not an article created by MSN, it’s a reprint of the Daily Star article I already mentioned — a British tabloid which is neither “legitimate” nor “mainstream” for my purposes. Sources matter to me very much, and even if tabloids get it right sometimes, tabloids’ reputations leave a lot to be desired. -rc
I work for what is now a major software development company as a Security Analyst. But when I started as a part-time contractor six years ago, I was one of the first members of an experimental group hired to do manual review of suspicious accounts.
We were the first, so there was no training to speak of. But we lucked out: we had time to learn before the great wave of spam, trolls, and AI arrived. And because our co-workers have encouraged us to learn a wide range of new skills, we are highly valued — enough that we are now full-time employees.
That minimum-wage stooge you’re angry with is probably a contractor, might not speak English as a first language, very likely got little training of any real value, and has to rely on scripts and possibly AI to make decisions — always very quickly. Do not blame the worker!
Put the blame where it belongs: on the owners and managers at ElevenLabs. They are ultimately responsible for what the company does and how it treats customers.
That manual reviewer who’s listening to what customers like Joyce Esser say in the privacy of their own homes? They might be sick shits who get off on it… but I doubt it. I don’t know anyone who does this job for kicks. And I might be wrong about the reviewers who work there — but among the long list of jobs they’re advertising right now, not a single one is for “Trust & Safety” (the professional term for that kind of work) or for any sort of content review specialist. I bet you they’re all contractors.
(And if they’ve got some sort of AI system doing the work, then blame the machine learning engineers, who are certainly making more than minimum wage!)
—
First a note: I allow Anonymous comments for good reasons, such as to make sure the commenter’s job isn’t endangered.
I take responsibility for not making it fully clear which actor in this drama was a “stick-up-their-arse minimum-wage stooge listening in … from 5,000 miles away” (aka “contractors”) and which may be higher-up folks at ElevenLabs itself (“employees”). Indeed the person who suspended the account was likely a low-level, off-shore, contractor type. Then Ms Esser contacted the company to appeal, and got the account reinstated, which was more likely to have been an actual employee, likely not “management”. Finally, the MIT-TR reporter contacted a management-level employee for comment, leading to “and was referred to their ‘prohibited use policy,’ though that doesn’t list any prohibitions on ‘inappropriate language’.” The result is still a mess, so I stand by the bottom line: the company is way, way in the wrong here, and by multiple levels of workers. As Anonymous makes clear, the first actor may not be culpable. -rc
The company is absolutely in the wrong. But as my teammates and I were discussing the story this morning on Zoom (I’m Anonymous, from above), the closed caption system transcribed “arse” as “arson” — which would have raised flags in our ML rules. So even AI is unlikely to do a better job.
In the end, only those who pocket the profits can make a real difference in the ways they treat customers.
p.s. Thanks for the clarifications on the sequence of events!
—
You’re welcome. It’s of course impossible to say if that mis-transcription was the culprit in this particular instance, but that raises all sorts of other problems. Even if the word was transcribed accurately (“Did you hear the Johnson’s house fire was arson?!”), it such should not result in automatic account suspension. If such was flagged, that means flagged for review, not for termination. Least of all when a medical device is involved. I know I’m preaching to the choir, but my point is that even if transcribed correctly, and even if it’s a “violent word,” context matters. 11L failed miserably here. Again, they need to step up their ethics, or spin this off to a proper stand-alone business. Yours has been the most informed voice of “consider the other side” (thank you!), but if anything, I’m more resolute after your arguments, not less. -rc
Is there some reason to think that a “medical device” was involved, let alone that the company knew that? She is using it for medical reasons, but that seems to be something she did without their knowledge.
The only references to “medical” that I could find on their web site seemed to be about providing medical advice verbally instead of hoping the patient can concentrate well enough to read. (And to read the often dense legalese that often replaces written medical instructions.)
To expand on what Justin pointed out, their web site is focused on “content creators”. Their pricing plans talk about how many different voices you are using, and their “try it out” links let you start by cloning an existing voice, which obviously won’t be your own.
Even for a voice you have “created”, you shouldn’t be able to put words in someone else’s mouth just because you trained an AI on recordings of other things that person said. The company probably doesn’t have a good answer for this, and just hopes it won’t happen too often — but part of risk mitigation is to limit what voices are used for unless/until they are sure that the person being imitated has agreed.
Joyce should be able to say what she wants with her own voice, trained to sound like her — but I doubt the company had any reason (prior to press contacts) to even assume that it was really her own voice, or that she was using for non-commercial reasons, let alone that she was using it in a medical device.
—
Unfortunately, the MIT article isn’t fully clear about it. They describe it as her using “an AI tool built by ElevenLabs”. They also make it clear that she has an account with the company, as this is what was suspended. It is at least reasonable to assume they knew it was her voice, as swearing to that was likely part of the set-up process. -rc
This is a sign about the direction technology is taking. And I really don’t like it.
More and more “they” don’t let you own or control things you pay for, you’re renting or you’re buying a license or there’s this ridiculous list of restrictions for something. Some of the worst is when it’s for something that doesn’t even require online functionality — like single-player games or devices where the network functionality is supposed to be just be gravy.
It’s aggravating at the least, but the story about ElevenLabs is this at its VERY WORST because it’s an invasion of privacy, it’s harming someone they’re supposed to be helping who has a serious medical problem, and they’re taking away a service that might be life-saving on a whim.
It’s bad enough when it’s a silly game service like Roblox where the censor/chat filter is so bad they might as well just disable chat. It’s certainly not protecting my kids in any constructive way I can discern, it’s clearly that way to protect the company from lawsuits. Someone else already ranted about YouTube and their Newspeak-ification.
It’s a symptom of a bigger problem. Companies want control because it makes them more money, it ironically lets them reduce their liability/risks (it shouldn’t!), it lets them be infantile pseudo-people entities that can do whatever they think of this week to make more money (this should be the new definition of “corporation”).
I do not like where this is going. Cyberpunk books were supposed to be warning of where things could go as well as entertainment, not “how to” guides for making megacorps.
That is called enshittification. The original term applies to online platforms, but anytime a company tries to extract extra money out of users while making the experience of using the product worse qualifies in my book.
For more on enshittification and what can be done about it, read this.
It’s long (about a half hour maybe to read) but really gives you the details of how this happens and what to do about it.
I’m not sure that this is pure evil, but then I often invoke Hanlon’s razor.
It is also true that there is a logical error on the part of the user and reader: one shouldn’t get mad at a person or a company for doing what *they* want instead of what you want them to do.
From our perspective it might be privacy violating or enshittification, but as long as they are transparent and don’t directly harm others, I think you have to let people do what they want to do. They get to choose how they want to offer their services, and it’s partly on us to choose valid services. The MIT article notes “But speaking through a device has limitations. It’s slow, and it doesn’t sound completely natural. And, strangely, users might be limited in what they’re allowed to say.”
Well, I’m not sure that’s strange, in the sense that they pretty clearly say in their terms of service:
——
2. Human in the loop
a) As provided in Section 3 above, use of our Services to provide tailored professional advice necessitates maintaining a qualified professional in the loop.
——
and nine “prohibited uses”, ending with “9. Do not use our Services in any manner contrary to ElevenLabs’ policies, purpose or mission.”
So if you spend ten minutes reading the terms of service, you can discover that this service wants has terms that would be unacceptable to most people.
The right answer, of course, is that the service go out of business for offering a crappy service. And as others have pointed out, 11labs isn’t a monopoly, there are lots of alternatives. Use market forces to drive services to the top instead of racing to the bottom.
This isn’t a story about an ALS patient that can’t say “arse”. This is a story about how as a culture we allow terms of service that are horrendous, and allow companies to get away with it because we can’t be bothered to be informed consumers. This problem goes way past consuming services. Google’s AI definition offers “Enshittification is an informal term that describes the gradual decline in quality of a service or product, especially online. It can also refer to the deterioration of society or politics.”
That’s the problem that needs to be highlighted and fixed.