ver the past year my biggest sites have come under various DDOS attacks. It is hard to guess who to attribute these to, but the matter of fact is, someone does not want these sites to be accessible. It is possible that different factions are independently behind these attacks, but since British police won’t lift a finger to help prevent this or even investigate this, I am left having to defend myself.

Blocking offending machines is sometimes possible, but it is very hard to block based on affiliation (or assignment) of an IP address, especially if the IP is assigned to some very large entity. Since Amazon controls so many IP blocks (even the A block of an IP address changes a great deal, not just the B block), it is very hard to block AWS servers that engage in DDOS attacks based on IP patterns, geography, etc. It makes AWS enormously dangerous and potentially destructive. It can easily take down online stores, causing huge financial damage and upsetting regular customers.

Several months ago it was alleged (albeit later denied by Amazon) that Sony had acquired access to AWS servers for the purpose of engaging in DDOS attacks, trying desperately to stop the distribution or leaks of its internal E-mails (now in Wikileaks, which Amazon banned after political pressure). AWS has a massive (over half a billion dollars) deal with the CIA and last week AWS also opened up to the NSA, which needs additional computational resources to spy on people and crack encryption. There’s no room for trust and AWS clearly disregards ethics, as a matter of policy.

There is a type of attack that has gone on several times now, meaning that it’s not some random incident and that it’s not going to stop unless serious action is taken, if not against the offender, then against the facilitator (host). This is why I decided to take it up with Amazon. I filled out the following form, having already lost many hours fighting DDOS (visitors were not able to access the site).

Here is a confirmation that they have received my report (around 7PM last night).

These are essentially DDOS attacks on Techrights, there is not much ambiguity there. Very large IP pools (not zombies PCs but AWS, unless Amazon’s servers too became zombies) are behind it, so I reported to Amazon that their systems are DDOSing my sites. They don’t even have “DDOS” as option. “Excessive web crawling” is the closest description to the activity, the others being completely irrelevant. Perhaps Amazon just views itself as incapable of doing such malicious things. How arrogant.

This is clearly a DDOS attack. This happened also less than one week ago (same attacker, same tools), for over an hour. It’s happening again this weekend.

Pattern of the attack: about 100 simultaneous requests (at the very same second) to many CPU-heavy pages, repeating in a cycle, taking down the httpd service, then waiting for the next round. Example request (time in US Western):

[27/Jun/2015:11:05:20 -0700] “GET /2015/01/ HTTP/1.1″ 200 331187 “-” “Typhoeus – https://github.com/typhoeus/typhoeus”

Typhoeus is always the Swiss army knife.

From Varnish we get the original IP addresses, e.g.:

  643 SessionOpen  c 54.225.55.182 46051 10.0.10.11:80
  643 ReqStart     c 54.225.55.182 46051 1557192107
  643 RxRequest    c HEAD
  643 RxURL        c /category/osdl/
  643 RxProtocol   c HTTP/1.1
  643 RxHeader     c Host: techrights.org
  643 RxHeader     c Accept: */*
  643 RxHeader     c User-Agent: Typhoeus - https://github.com/typhoeus/typhoeus
  643 VCL_call     c recv lookup
  643 VCL_call     c hash
  643 Hash         c /category/osdl/
  643 Hash         c techrights.org
  643 VCL_return   c hash
  643 VCL_call     c miss fetch
  643 FetchError   c no backend connection
  643 VCL_call     c error deliver
  643 VCL_call     c deliver deliver
  643 TxProtocol   c HTTP/1.1
  643 TxStatus     c 503
  643 TxResponse   c Service Unavailable
  643 TxHeader     c Server: Varnish
  643 TxHeader     c Content-Type: text/html; charset=utf-8
  643 TxHeader     c Retry-After: 5
  643 TxHeader     c Content-Length: 419
  643 TxHeader     c Accept-Ranges: bytes
  643 TxHeader     c Date: Sat, 27 Jun 2015 17:56:01 GMT
  643 TxHeader     c X-Varnish: 1557192107
  643 TxHeader     c Age: 1
  643 TxHeader     c Via: 1.1 varnish
  643 TxHeader     c Connection: close
  643 Length       c 419
  643 ReqEnd       c 1557192107 1435427760.530639887 1435427761.228856087 0.219283819 0.698100090 0.000116110
  643 SessionClose c error
  643 StatSess     c 54.225.55.182 46051 1 1 1 0 0 0 257 419

This is clearly abuse as it amounts to about 100 requests per second lasting for a long period of time.

I told Amazon: “Deal with it as soon as possible and keep me informed or I will take action against Amazon for harbouring DDOS attacks.”

The attacker’s IPs vary too much to block, but they are all AWS (54.225.55.182 and many others).

Amazon has not even responded to acknowledge that they’ve received my abuse report. It had been about 14 hours and I was still waiting. There was nothing to even say they’re looking into it. They are clearly not taking disciplinary measure and security measures, perhaps not employing staff in the weekend to deal with abuses by their network, only to their own network (when it’s them who are the victim).

Amazon is a malicious, irresponsible company.

14 hours after my report they finally got back to me and even then their response was weak and useless. Here is what they wrote:

Hello,

Thank you for contacting Amazon Web Services. We take reports of unauthorized network activity from our environment very seriously. It is specifically forbidden in our terms of use.

Because Amazon EC2 Public IP addresses may change ownership frequently, without additional information we will be unable to identify the correct owner of the IP address for the period of time in question.

So that we can process your report and identify the actual customer in question, we require the following additional information. Please note that we will not open attachments under any circumstance.

* Destination Port and Protocol
* Intensity and frequency of activity in short log extracts, no larger than 4KB

For a faster response, please file your report using the AWS Abuse form at the link below:

http://aws-portal.amazon.com/gp/aws/html-forms-controller/contactus/AWSAbuse

We appreciate your help in providing the necessary information requested.

Best regards,
Amazon EC2 Abuse Team

I am going to follow this up with updates.

Based on the time of Amazon’s response, around 9AM UK time, Amazon is just trying to save money and thus increase its profit (while famously evading tax in the UK) by strictly working from 9 to 5. Abuses like these require a dedicated 24/7 team.

Update: I had to spend more time reporting this to Amazon again, this time with stronger language:

You must take action immediately and tell me who is behind these attacks (your IP addresses). I am utterly disappointed with you taking 14 hours just to respond (and not take action against the attackers). I wrote about this with more details in https://schestowitz.com/Weblog/archives/2015/06/28/aws-ddos/

You have asked me for:

* Destination Port and Protocol
* Intensity and frequency of activity in short log extracts, no larger than 4KB

I already gave you those! But you are ignoring my report or failing to investigate, so I am adding yet more evidence from the Apache back end.

Please respond immediately and tell me who the perpetrator is so that I can take action. Failing to do so, I shall take action directly against Amazon.

Here is a screenshot of the form (they refuse to correspond by E-mail, so I have to refill their tedious forms each time).

Update #2: Almost 24 hours later and there is no breakthrough, just more waste of time. Here is what they wrote again, merely repeating themselves:

Hi Roy,

I apologize for the inconvenience caused from this attempt on your website.

I do understand the frustration and I am sorry that this has happened. I have checked with our Abuse team once again and they would indeed need additional information to continue their investigation. This was mentioned in their reply on the Abuse case that currently exist in the system.

The Abuse team will continue to investigate and take action against this, but they would need the additional information:
* Destination Port and Protocol
* Intensity and frequency of activity in short log extracts, no larger than 4KB

Once you have that information, please provide it by responding to their e-mail: ec2-abuse@amazon.com and including your case number:
55322041040

Best regards,
[redacted]
Thank you.
Amazon.com

Here is my response:

Hi,

I don’t understand why it is “Message From Customer Service” because I am not an Amazon customer, I am a victim of Amazon customers. I need to speak to technical people and the issue must be rectified as soon as possible. I need the names of people at Amazon who are dealing with the case and the Amazon customer who is the culprit in case I file a lawsuit. As for the additional requirements, which you sent to me twice despite me providing them already twice when I filled out an abuse report, what else do you need?

You say:

* Destination Port and Protocol
* Intensity and frequency of activity in short log extracts, no larger than 4KB

I provided these twice and even additional logs the second time (about 4KB). I need you to carry out an investigation as this is your job, not mine, and I don’t have the tools to figure out which of your customer uses your facilities to do this. Amazon is facilitating DDOS attacks and I demand to know who is dealing with my case and for further action to be taken immediately. You also have my phone number (since the time of the incidents) and you should have phoned me last night, not replied by E-mail 14 hours later.

Update #3: Amazon finally found the source of DDOS (its own servers abusing others) — 24 hours after I reported it! Very slow a reaction time.

Here is what I got at 3:36PM:

Dear Abuse Reporter,

Thank you for submitting your abuse report.

We’ve determined that an Amazon EC2 instance was running at the IP address you provided in your abuse report. We’ll investigate the complaint to determine what additional actions, if any, need to be taken in this case.

If you wish to provide additional information to EC2 or our customer regarding this case, please reply to ec2-abuse@amazon.com with the original subject line.

Thanks again for alerting us to this issue.

Case number: 55464910873

Your original report:

* Source IPs: 54.225.55.182
* Abuse Time: 2015-06-27 18:05:00.0

My response is still strong in tone, having suffered enough already:

Hi,

Please provide me with information about who the client is and the nature of the account that did this. It is imperative that I know this because DDOS attacks on my sites have been a major issue for quite some time and we have only speculations about who or what may be behind it. Hundreds of hours have been lost due to DDOS attacks this year alone. There was also damage to databases and hence partial data loss.

Additionally, please provide me with names of people in Amazon who dealt with this case. 24 hours response time to a DDOS report is totally accessible, especially for a multinational company like Amazon that operates massive datacentres with capacity to paralyse the world’s biggest operations. I shall be documenting this case in my blog and will pursue this as far as necessary.

Regards,

Update #4 (29/6/2015): Amazon has still not replied to my message. Instead I got some sort of public relations from them (this morning):

Hello Roy,

I do apologize for your emails being inappropriately addressed.

I do see that our dedicated technical abuse team has contacted you advising that the abuse case is being investigated and should you wish to provide or require any further information, that team would be the best to contact.

Please contact them at ec2-abuse@amazon.com.

I hope that this is helpful.

Thank you for your inquiry. Did I solve your problem?

If yes, please click here:

http://www.amazon.com/gp/help/survey?p=A3DTXBBD5H1UEV&k=hy

If no, please click here:

http://www.amazon.com/gp/help/survey?p=A3DTXBBD5H1UEV&k=hn

Best regards,

[redacted]

http://aws.amazon.com

I responded as follows:

Hi,

I am actually waiting for response from them. I have been waiting for over half a day now. Can you please rush them. I need to know who is behind the attacks (which Amazon customer) and who in Amazon deals with this case.

I also followed up with the other team:

I have been waiting for a response for nearly a day now and it seems like stonewalling. If you do not reply by the end of that day, I may pursue legal action against Amazon, whose facilities are being used against my site while Amazon remains unhelpful and unresponsive.

It turns out, in the mean time, that the EPO uses AWS and some suggest that I report the latest DDOS attacks to the FBI (where the server IPs are based).

Update #5 (30/6/2015): Still no real progress. It has been a long time since the second attack and Amazon gives plenty of reasons to take legal action.

I apologize for the late reply on this. If you could please respond with your abuse case number, I would be happy to reach out to the team on your behalf to expedite the process!

Let us know if you have any other questions or concerns. Thanks!

Thank you for your inquiry. Did I solve your problem?

If yes, please click here:

http://www.amazon.com/gp/help/survey?p=A31TVRCH796UWU&k=hy

If no, please click here:

http://www.amazon.com/gp/help/survey?p=A31TVRCH796UWU&k=hn

Best regards,

[redacted]

http://aws.amazon.com

This was a useless respond, so I gave the reference number:

EC2 Abuse Report [55464910873]

I expect a reply by the end of the day.

Hello Roy,

I hope this email finds you well.

In the event that you have not provided your abuse case number for us to follow up for you on, please kindly reply directly to ec2-abuse@amazon.com. I can see that you have been corresponding to aws@amazon.com.

Please include the abuse report number you wish to address in the subject line of your email.

I hope this helps.

Thank you for your inquiry. Did I solve your problem?

If yes, please click here:

http://www.amazon.com/gp/help/survey?p=ARC3CZ1A921WW&k=hy

If no, please click here:

http://www.amazon.com/gp/help/survey?p=ARC3CZ1A921WW&k=hn

Best regards,

[redacted]

http://aws.amazon.com

So Amazon failed to send the detailed requested by the end of yesterday, just wasting more time with a useless reply again. I sent the following more strongly-worded reply:

Hi,

I have received not one but two absolutely useless (lacking substance of any kind) replies from you. You wasted more time instead or responding to my message which said:

———-
I have been waiting for a response for nearly a day now and it seems
like stonewalling. If you do not reply by the end of that day, I may
pursue legal action against Amazon, whose facilities are being used
against my site while Amazon remains unhelpful and unresponsive.
———-

If you do not respond in the next few hours with names of people at Amazon who deal with this case (EC2 Abuse Report [55464910873]) and the identity of the offending account that has been attacking my site on numerous occasions, I will prepare a lawsuit against you, Amazon, first for launching DDOS attacks from your servers and second for not obeying my request to know who is attacking my site. I may, additionally, contact the FBI to investigate, as the IP addresses in question are based in the US and DDOS attacks are a federal crime. I give you 2 hours from now to respond to my original request.

This entry was posted on Sunday, June 28th, 2015 at 9:01 am and is filed under Internet. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.