From True’s 17 October 2010 issue.
On the whole, This is True readers are a pretty technically savvy bunch: many of you use “tracking” email addresses — addresses which readers have used only to subscribe to my newsletter(s) — and I’ve had a number of reports this week from readers who have received spam to those unique addresses. That’s obviously a big concern to me.
I use the industry leader “Email Service Provider” (or ESP), AWeber, for all of my lists (except, for technical reasons, the Premium — paid — list). I’ve used them since 2006 for “blog notification” email subscriptions for my personal blog, Cranky Customer, Jumbo Joke, the True Stella Awards, and several other minor lists. I moved This is True there in December 2009, when I started having significant service problems with the former industry leader, Lyris.
A Constant Target
Like all ESPs, AWeber fends off thousands of attacks on its servers every day (literally: every day) from criminals trying to get hold of their vast store of email addresses used by thousands of customers. Earlier this year, one of those criminals succeeded in obtaining subscriber lists from AWeber, including a portion of my subscriber list.
This week, these criminals apparently got my entire list (along with many others).
On both of these occasions, I’ve communicated directly with the CEO of AWeber. As with every email publisher (like me), their reputation is key: we’re asking you to trust that we won’t abuse your address. You sign up to get a newsletter or other service, not spam. And I hold that trust as sacred — even though you’ve asked to get information from me, I never send ads-only mailings; that’s not what you signed up for. So I’ve obviously put a lot of trust into AWeber, that they have the highest integrity too.
That includes my having trust in their security. “Industry standard” security isn’t enough: they must be industry leaders and have their lists buttoned down tight. But with literally thousands of intrusion attempts per day, that means keeping on top of things, and AWeber has a full-time security team — doing nothing but checking and tightening security anywhere there’s even an inkling of trouble.
Doing the Right Thing
But last week, the criminals succeeded. AWeber did the responsible thing: they immediately acknowledged the security breach in public, on their blog, signed by the CEO himself.
While I’m obviously concerned that this security breach affected me, I’m much more concerned that it affected you: you undoubtedly have more spam on the address(es) you entrusted to me because of this. And for that you have my apologies.
If you are using a “tracking address,” you have the advantage that you can shut it down: the “change subscriber options” link at the bottom of each free edition newsletter not only allows you to easily unsubscribe, but allows you to change your delivery address too. If you can easily shut down your old address, I do urge that you do so now, and I hope you will continue to get This is True on a new address.
Can I Guarantee This Won’t Happen Again?
Fighting spam and spammers’ tactics is an ever-escalating battle. Spam is now the vast majority of all email traffic for one simple reason: it makes the spammers money. They lie, they steal, they defraud, they infect your computer with malware to steal your bank account information — anything they can do to rip you off so they can deprive you of your hard-earned money.
They’ll literally do anything to steal more — including invest their ill-gotten earnings in programmers and network professionals to do anything they can to expand their activities.
It’s disheartening that AWeber’s security was breached. I’ve learned a lot about their security procedures after signing a Non-Disclosure Agreement (or NDA), and I’m impressed with the investments they’ve made in their security infrastructure — the specifics of which I cannot repeat due to that NDA. I’m convinced that moving to another ESP will not result in better list security, so I have chosen to remain with AWeber.
Yes, as industry leader, they’re a juicy target. But, as industry leader, they can also afford to fight back. Other ESPs are being targeted, and I have no doubt other providers have suffered such breaches too. (Lyris, did, while I was using them, in a highly publicized incident. But the criminals didn’t get my list that time.)
Reputation Management
I’ve spent more than 16 years building my own reputation — as one of the first email publishers in existence, as an anti-spam educator, as a successful online businessman. There is no perfect security, but I am confident that AWeber is doing everything that can be done, and will continue to improve as this fight evolves.
So I think it’s a “big deal” that they still have my confidence.
Might criminals succeed again in getting your address from the lists you subscribe to, from your own address book, or from your friends? Yes. The only way to be completely safe is to not be online at all. Just like the only way to not be robbed is to not go outside your home at all. But then, robbers (and spammers!) can just come into your home, too.
They try daily, and they will continue to succeed from time to time. The only thing you can do is be smart and careful, perhaps change your address now and then (and virus-scan your computer regularly!), because they will succeed again — if not at AWeber, then somewhere else.
It’s a War, After All
And I’m using the word “criminals” advisedly: there is no other word for them. I know AWeber is working with the FBI on this, and I truly hope the FBI takes this case seriously. This is organized crime, and it does have serious repercussions on a significant business sector — one of the few that is still growing in this horrible economy.
I believe the “spam war” is a real war, complete with collateral damage and innocent victims. But it’s a war we must win if email in particular, and online business in general, is to be viable economic force.
Again, my apologies for any “collateral damage” that you’ve suffered.
– – –
Bad link? Broken image? Other problem on this page? Use the Help button lower right, and thanks.
This page is an example of my style of “Thought-Provoking Entertainment”. This is True is an email newsletter that uses “weird news” as a vehicle to explore the human condition in an entertaining way. If that sounds good, click here to open a subscribe form.
To really support This is True, you’re invited to sign up for a subscription to the much-expanded “Premium” edition:
Q: Why would I want to pay more than the minimum rate?
A: To support the publication to help it thrive and stay online: this kind of support means less future need for price increases (and smaller increases when they do happen), which enables more people to upgrade. This option was requested by existing Premium subscribers.
I know I am appreciative of the efforts you go through to help keep the SPAM to a minimum. For your users, I thought I’d throw out a couple things that have helped me in my little corner of the war, starting with an obvious one:
– Read Randy’s Spam Primer and make sure you tell your family and friends to do the same.
– Consider GMail for mailing lists. Their SPAM filters have been amazingly accurate for me over the last few years. In general, no more than 3-4 misidentifications per year (SPAM in Inbox or real message to the Spam folder).
– Check out Thunderbird if you use a local email client (from the folks who make Firefox). I’ve been using it for about 3 months now and it’s adaptive junk filter is getting better all the time. Initially, about 50% of the SPAM got to my Inbox, but now after 90 days, it’s running closer to just 5%-10%. I’m sure by the end of the year, it’ll be even better.
Thanks Randy. I appreciate the extensive information.
I am on another list whose provider got compromised sometime in 2009, and the owners of that list never bothered to respond to multiple queries about it, except for a single, brief answer along the lines of “we’ll be sure to ask someone in our technical department to look into it”.
For those with a GMail address, you can setup a pseudo targeted email address by appending “+uniquephrase” to the username portion, such as example+phrase @ gmail.com. Combined with filters, it’s an easy way to organize email.
And if you’ve already got a targeted email address and don’t want to change it, setup a filter in GMail using doesn’t contain “This is True”, and set the message to be automatically deleted. Since all the official email is sent with the name “This is True”, the filter *should* catch and delete all the spam automatically.
And, if you’ve got a targeted email address and want to change it, at the aweber control panel, click the “edit contact information” and change your address there. No need to work through the whole unsubscribe + resubscribe process.
I’ve been using the email+comment@domain method of generating targeted email addresses for years. It’s worked with all 5 of the ISPs I’ve had over the years. It works because it’s a basic requirement in the RFC standards for email.
The one problem I’ve run into is the numerous online forms that use defective email address validation and incorrectly reject email addresses with a + in them.
I was very pleased when I first signed up for Premium This is True and it accepted my targeted email address without complaint!
The email+comment @ domain.com email address works great with any system having a properly compliant email handler on it. Like David in Berrien Springs, I’ve had a few mailing lists improperly reject it, which I don’t have a huge problem with. I do have a huge problem with sites that accept it, but then won’t handle that format when trying to unsubscribe! I’ve been trying for almost a year to unsubscribe to a mailing list on one particular site where their only method of unsubscribing is via an automated unsubscribing link, which loses the ‘+’ in the address in the unsubscribe request and leaves me subscribed. Numerous subsequent emails to their support people have gone unanswered, and I remain annoyingly subscribed to the email list. My final attempt to unsubscribe will be via an email to the CEO of the company. So the targeted email addresses are useful, but do have an occasional annoying drawback….
—
I’m quite sure that doesn’t happen with AWeber. But if it did, I’d get on the phone with the CEO instantly, and ensure it was fixed within the week. -rc
David beat me to the punch in mentioning “the numerous online forms that use defective email address validation and incorrectly reject email addresses with a + in them.” The one which I bothered to track down came from /html-form/javascript-form-validation.phtml at http://www.javascript-coder.com — Randy, is there any chance that you or someone you know has enough clout to get them to fix it? I sent them a message myself, but I don’t think they’ve acted on it.
—
Sorry, but I don’t have any contacts there. Maybe they’ll get a “Google alert” on their domain and see your request, though! -rc
Don’t sweat it, Randy. Thanks for the heads up, but seriously, don’t beat yourself up. We like you, we know you’re a good egg. Enough cliches now. Carry on.
—
I’m not killing myself over it, but yeah: I sweat it. And I wouldn’t have it any other way. -rc
Of course of us with gmail addresses didn’t even notice.
Gmail spam filters rock.
I have a healthy dose of skeptical when anything that smells of spam comes down my pipes. If I didn’t ask for it will delete it.
I WILL remain entertained and amazed by This Is True.
Security breaches happen a lot more than people know and this is yet another skirmish in the war.
And, yes, it is a war.
Since this goes through the Gmail servers, there’s already a filter in place – but that does explain the brief spate of spam that snuck in the other day. Thanks!
THAT explains the unbelievable number of spams I got this week. Glad I know what the problem is. And I’ll just delete them…. doesn’t bother me at all.
Randy, thanks for telling us.
What’s a couple of extra pieces of spam so long as I am receiving This Is True!
The good news is that TIT is run by someone with the manners to let us know it has happened! And Mr.Cassingham should be relieved to know that those with the brains to get TIT (should!) know spam when they see it!
Thanks for the quick notification. I didn’t see it on Monday’s Premium edition. I get stuff from other sites via AWeber and haven’t heard anything from any of them (yet).
I hadn’t even noticed until this arrived, but there are more than usual spams in my folder – five! – and none got to the inbox. Google rules!
Keep on with the great work you do.
I subscribe using an address only used for email lists and not posted to the web anywhere. For the first time ever I received two spam messages at this address that my provider dumped into the “known spam” folder.
Thank you so much for telling us this. Luckily I’m with hotmail, so most of the spam will get filtered away from my inbox. It’s reassuring to know that you will keep us informed if something like this happens. Thanks again.
Only had one spam mail get through into my inbox this week. I did have a spate a couple of weeks back, so I guess another server got compromised.
I use GMail too…it’s generally pretty good at catching spam. I very rarely have spam come through to my inbox as a general rule.
—
A spam surge doesn’t mean that a server has been compromised — I was seeing a huge surge before this happened. It’s another skirmish in a long, long war. -rc
We haven’t been bothered with more than one spam per week for a few years! The secret: Google mail. When I occasionally look at what they stopped, there’s a pile of stuff in there. But almost none of it gets by them; and very few “false positives” — that is, good email mistakenly labeled spam.
—
The way I use Gmail to filter my thisistrue.com email is detailed here. It’s fabulous. -rc
I’m a private white-hat fighter of spam and viruses. I read what you wrote and agree with most of it, but with one glaring point of departure. That point by itself would be a deal-breaker for me as to AWeber. And it leaves my anger over the matter of their breaches unresolved.
Why should there even be a way to access the entire database from the broad Internet? How can their “security infrastructure” even allow that? Certainly individual list owners, such as you, have to be able to see your lists from what I take it to be secure connections. But no command to see all lists should work from the WWW, I’m sorry. That is kindergarten stuff security-wise; I don’t need an NDA to know otherwise.
AWeber’s experience with the highly damaging break-in from early in the year should have been enough to get it to change its model, rather than continuing with the design vulnerability.
Finally, of the perhaps 30 discrete tracking addresses of mine that were leaked the last time, only two companies (email list-service users) bothered to inform their subscribers in chagrined tones about what happened. The others? I have meanwhile disassociated myself with many of them, or at least communicated my displeasure at their silence over the matter. (So thank you very much for being public about what happened — as I knew you would be!)
In closing, I’ll ask: What if the IRS or Social Security Administration made the architectural-security goof that AWeber has — now twice? I am given to understand those agencies’ computers and networks are in abysmal shape. It must be that they don’t allow access to the whole shebang, though, or such would have happened there long ago.
For those with a GMail address, you can setup a pseudo targeted email address by appending “+uniquephrase” to the username portion, such as example+phrase @ gmail.com.” –Kyle et al.
Won’t some of the spammers figure out that they can strip the “+uniquephrase” and it will still get delivered (unless folks implement filters to strip the raw addresses?
Kudos to Randy and AWeber for being upfront with the bad news.
Stuff happens. My spamblockers have earned their keep this past week or so. Wondered why.
As the kids say; stuff happens. No worries. If it didn’t happen to you it would have happened via another contact.
Thanks for the the good humor you send gratis. When I became disabled I lost a great deal and the ability to laugh was one of those things. You have restored my sense of humor. Thank you.
—
You’re very welcome, Bonnie. I’m very happy to have helped you. -rc
At least Gmail has a nice spam filter but I’ve noticed a lot more spam from this week.
Glad to know where the problem comes from.
No worries, most of the major email providers have spam filters that will have their messages flagged as spam in no time. I didn’t really notice much more spam than usual. It’s an ongoing battle and probably will be for the rest of our lives. The only way to stop it completely would be for the few gullible people who respond to the emails.
This week I’ve received 3 emails purporting to be from Adobe encouraging me to click a link to update to their newest version. This may have nothing to do with the stolen list but I do worry about the uninitiated when such believable emails are received and acted upon.
I also received an invite from a nice sounding Russian lady but politely declined by shoving her in the trash folder.
—
The “click here to upgrade” scam is to get you to install software on your computer that allows remote criminals to take it over. Then what? Then they can use your computer to send spam, or watch when you type your user name and password into your bank account, or any other similar scheme that trust me, you won’t like. We have to think before doing. If some random criminal sends a spam email to you trying to get you to do something, just remember that action is in their interest, not yours. -rc
Actually, if both the status page (the mentioned “change subscriber options” link) and the adress change postback go over the wire unenrypted then I’d NOT trust them to be security-conscious enough. So they care enough about someone hacking into their servers but they don’t care about any old hacked router outside their control picking off addresses from passing packets? Or are they just too cheap to pay for a properly signed SSL certificate?
Maybe I’m exaggerating, but then maybe again someone with a little clout should ask them.
—
If your home router is hacked, you have worse problems. In general, email is not encrypted — every time you log in to get your mail, unless that connection is specifically encrypted, you’re not only revealing your email address to your compromised router, but also your email’s password. I think using a non-SSL link to unsubscribe is hardly something of concern. -rc
Dunc in Alberta asks, “Won’t some of the spammers figure out that they can strip the “+uniquephrase” and it will still get delivered (unless folks implement filters to strip the raw addresses?”
This is exactly why I use address+unique for *all* my legitimate contacts, leaving the spammers nonplussed.
I am much less concerned about errant Spam” than of someone being able to access personal data from my computer. Have you and A Weber discussed this possibility?
—
No, since there’s nothing whatever about getting your email address that will help anyone do that any more than any random spam. Let’s not blow this out of proportion: having someone learn your email address does not give them access to your computer! -rc
Thank you for letting us know about this problem. Not many sites would do that.
I was very surprised to be getting spam all of a sudden at this email address since I don’t give it out to many people. Never mind. Now that I know how they got it I can fix the problem.
You know I never cared much about Spam till I saw this documentary where they showed how spam funds terrorism, child pedophilia, assassinations and dictatorships. If that documentary was telling even a grain of truth then I would like to think law enforcement agencies would be a lot tougher on spam.
Your security problems coincided with the Tech people at work fiddling around with the spam filter so I just made the assumption that it was something they did which led to a small uptick in my spam. I do trust you to make good decisions about the services you use and be diligent about protecting your readers from spam. But stuff happens. This gave me a chance to get reacquainted with all my lost relatives who have died leaving me their fortunes!
Thanks for keeping us all up to date.
Thanks for the notification, but I can see only one way that this will affect me: I’ll have to empty my SPAM folder a little more often, definitely not a big deal.
And you are right, the only way to avoid this is to stay off the internet.
What a shame this happened. But, unfortunately, this sort of stuff happens. Other large trusted companies have been breached where the information was more personal, such as credit card and social security numbers. I don’t blame This is True, and hardly blame the List Management Company.
Who I do blame is all the dumb-asses that create a market for spam by purchasing from spam ads, or falling for spam scams. If more people were educated on this, spam would shrivel up and die. It only takes a few idiots to make spam and scams profitable for the criminals.
PS – I haven’t noticed any real increase in my spam to this address, so hopefully it’s not that much of an issue for most of your readers.
Dallman, Germany who posted on October 23, 2010 is right. Also, why isn’t all the data encrypted???
—
Because the list owners need to be able to search through the data base. -rc
You are a person of integrity, and I trust you and your judgment. To use the internet at all is to be somewhat vulnerable to these despicable criminals. We do the best we can to protect ourselves, and in that regard your advice over the years has helped immensely. Meanwhile, they do the best they can to sabotage any attempt to stop their illegal activities. It will continue to be a battle until we find better ways to balance safety for us against government intervention (which, after all, would be even less secure). I have no doubt that that day will come. Meanwhile, thank you for your honesty and caring.
I work in the IT industry and have worked on high security gateways, and I can tell you that one of the things that is known but NOT accepted in the industry is that you can NOT keep ALL the bad guys out ALL the time. You work hard to keep everything up to date and be proactive in dealing with new attacks, but that does NOT cover everything you don’t yet know about.
I’d rather you stayed with and used a service that are quick to identify being attacked, respond to it, and clean up, and then (most importantly) let their clients know what’s happened. Way too many organisations like AWeber do NOT tell the clients when they get breached, so you can’t do anything to clean up at your end. With AWeber being open and honest, you have a chance to preempt most of the spammers and clean up from both ends at once. They are good and doing good, you are doing good, so I agree, stay with the good guys.
My email has a whole series of anti-spam software at the host service and on the system. They work so well I rarely see spam, and never a second from the same source after I’ve identified it to the system. I’ve not noticed any increase. My logs show my average of only one new spammer this month, about a week ago.
Thanks for being quick to let us know, and please thank your service provider for being open, honest, and quick to let you know.
The use of “+” for sub-addressing and filtering is a convenience provided by some mail systems, rather than an RFC-requirement. Other systems may use other characters, such as “-“. No one should rely on it working without testing it, and no one should be surprised if they try to email to such an address and it doesn’t arrive.
By contrast, although it supports “+”, Gmail ignores “.” altogether in mailbox names, so “this.user @ gmail.com” is the same as “th.is.us.er @ gmail.com”.
What the RFCs do say is that a system MUST NOT refuse to accept as valid any mailbox name that does include a “+”.
See this Wikipedia article for more information.
In spite of the number of breaches, people still think that “computer security” can be achieved. (I put the phrase in quotes for the same reason that “Tooth Fairy” would be in quotes; both are imaginary.)
The worst abuses are currently in the news: computerized voting irregularities. Anybody who thinks paper ballots can be improved by a computer is not someone I can respect. By the way, that’s the mildest way I can state my negativity toward computer voting. Feel free to amplify it in your own mind.
@rc – No, I didn’t mean my home router, and I do think mail providers not supporting some SSL variant on all protocols should be boycotted, at least I do. The point is, the benefit/cost ratio of using https in this case is good because the cost is negligible (in relation to the size of the user base).
—
I do agree that end-users’ mail should be sent and received from their computer to their provider via an encrypted link. Just as I agree that there should be a secure and identity-verified way to exchange mail between servers. We’re not there yet: the norm is for it all to be unencrypted. Until the powers that be set up a truly standardized encrypted and identity-verifiable protocol, though, we will continue to have problems. -rc
I read the explanation on the AWeber CEO’s blog and was left wondering, because the breach happened on a Saturday and wasn’t discovered till the following Monday, if AWeber cheaped-out and didn’t employ weekend technical support. If that’s the case, then they’re partially culpable.
—
Not knowing details of the break-in path, I don’t know when it might have become apparent, so I have no way to judge whether they “should have” been able to detect it sooner or not. -rc
My email provider has a SPAM filter that’s (IMO) about 90 percent accurate. I’ll have maybe one or two a week that sneak through or one or two that get flagged that are not SPAM, but I can flag them and not have that problem again with those messages.
What’s annoying, though, is I get 3-4 messages a week from VistaPrint and if I flag one, they all go to SPAM even if I want to keep just one message for the special offer included.
—
Here’s a question to consider: if you consider a company’s mail spam, why in the world do you want to do business with them? You’re simply encouraging more spam. Find another vendor who won’t abuse your mailbox. -rc
I knew something had happened when the amount of new spam increased greatly. I consider myself very lucky that AOL is my email server and between COX communications and my son’s computer expert I have several excellent layers of protection.
Thank you for being open & honest with us. Imagine if we had that kind of communications from our elected leaders and/or BP??
Well I appreciate you letting all of us know. Today I checked my email and I had about 50 spam messages when normally I have none so I am thinking that is why. I thought it was kind of odd but now it makes sense. I understand that nothing is fool proof though. I am pretty sure my email addy has already been sold several times over anyways. I am glad that gmail has such an awesome spam filter.
I do not know if these two things are related but I had an episode of blue screen death shortly after this notice was posted. I am not a techo-girl but I did know how to access safe mode and then restore.
I have since come to suspect that “restore” is the computer shop’s favorite app.
Well, my geek-boy managed to save all my files (no, of course I did not back any of them up), reinstall, defrag, update and create an antidote to the blue screen flu.
I am back!
Stuff happens, it is over and my geek-boy lives across the street and works for free because I told him he is in the will. Life is good.
—
I doubt there’s any correlation, but glad you had a Geek Boy to help! -rc