About Me

Curriculum Vitae

A brief list of my current skill set

Bloggybits

The Git Cheat Sheet
Friday, 6th September 2013, 11:30

for github, bitbucket, that kinda stuff

CoffeeScript and TypeScript are Stupid
Saturday, 17th August 2013, 11:21

Don't use them!

Changing the Order of the jQuery Event Queue
Wednesday, 3rd July 2013, 20:27

It's just a push to the left

How Do Spammers Get My Email Address?
Wednesday, 15th May 2013, 18:03

I think these days I have a pretty good idea

XSLT, node.js 0.10 and a Fun Two Days of Native Modules and Memory Leaks
Thursday, 25th April 2013, 17:14

documentation makes things less cryptic, so lets not write much of it

Fixing CentOS high cpu usage when running as a virtual machine under VirtualBox
Sunday, 21st April 2013, 20:28

innotek rocks! I mean Sun... I mean Oracle...

Repairing a dK'Tronics Keyboard and Scoping Out a ZX Spectrum 48k - Part One
Sunday, 17th March 2013, 23:51

What signals inside of it actually look like

Tabs vs Spaces and Why You Should Always Use Tabs
Monday, 4th March 2013, 19:51

Spaces are bad, just real bad

Why you should ban Amazon's Cloud IPs
Thursday, 27th December 2012, 14:50

And how to do it in nginx, Merry Christmas Amazon

Building Better jQuery DOM Inserts
Thursday, 20th December 2012, 15:18

Break it down baby

SEO Companies - Don't Waste Your Money
Wednesday, 12th December 2012, 16:16

Spammers by any other name

Pulse Width Modulation and How 1-bit Music Works
Wednesday, 5th December 2012, 23:34

Beep beep multi-channel!

Making PDFs from HTML on your webapp in CentOS
Thursday, 29th November 2012, 14:00

Not as easy as it should be

Some Days I Wish For an Async String.replace
Monday, 19th November 2012, 12:59

MinnaHTML to the rescue!

Object Oriented Programming for Javascript Dummies - Part 2
Tuesday, 6th November 2012, 15:33

OOP it up to the OOPballs

Projects and Sillyness

MAME Cabinet Diary

How I built my own arcade cabinet

Loading Screen Simulator

I don't miss the ZX Spectrum, I still use it!

The Little Guy Chat Room

It's a Pitfall inspired chat room

GPMad MP3

A fully featured MP3 player what I wrote

GP Space Invaders

My first little emulator

GP32 Development Page

Some info and links about this cute little handheld

Disney Nasties

Uncensored images, you must be 18 to view them

Minna's World

The cuttest fluffy puppy in the world

Diary of a Hamster

Learn about how hamsters think, first hand

Utilities

Time Calculator

A simple little online utility for working out how many hours to bill a client

A Few Links

Why you should ban Amazon's Cloud IPs
Thursday, 27th December 2012, 14:50

Life used to be simple, you made a website, submitted it to Yahoo and maybe a load of other search engines via a tool or website. These days, in some respects things are simpler still, you submit it to Google if it hasn't found it already, and Bing if you are feeling generous, then you sit back and get scraped by sites like Pinterest who take your images and serve them as their own.

But Pinterest isn't the only problem, there are loads of other similar start-ups out there doing the same, many you won't have heard of, many you'll never hear of. And Amazon's cloud services appear to be the system of choice for them to use.

Added to this, there will be a number of unscrupulous... if you don't consider the above as such... people who will abuse these same cloud services to scrape your content and generally try to free-load off your output whilst eating up your bandwidth.

The solution? Just block Amazon's IP ranges from your websites. Don't use a firewall, this stops you accessing services on AWS, just block them from reading your websites, then you only have to worry about Russian and Chinese scrapers. :)

Blocking Amazon Cloud IPs in nginx

Firstly, you need to create a new file to put the block list in, let's call this blocklist.conf, and we need to make sure we include it in the http section of our nginx.conf file. It shouldn't matter where, so if you use a wild-card include that picks up *.conf in a sub-directory, you can just add the blocklist.conf there and not do the next bit.

    http {
# all your usual generic stuff here
include blocklist.conf;
}

And now for what we'll be putting in our blocklist.conf file, which will use the IPs from the official sticky forum thread on the Amazon AWS forums:

    # US East (Northern Virginia)
deny 72.44.32.0/19;
deny 67.202.0.0/18;
deny 75.101.128.0/17;
deny 174.129.0.0/16;
deny 204.236.192.0/18;
deny 184.73.0.0/16;
deny 184.72.128.0/17;
deny 184.72.64.0/18;
deny 50.16.0.0/15;
deny 50.19.0.0/16;
deny 107.20.0.0/14;
deny 23.20.0.0/14;
deny 54.242.0.0/15;
deny 54.234.0.0/15;
deny 54.236.0.0/15;
deny 54.224.0.0/15;
deny 54.226.0.0/15;

# US West (Oregon)
deny 50.112.0.0/16;
deny 54.245.0.0/16;
deny 54.244.0.0/16;

# US West (Northern California)
deny 204.236.128.0/18;
deny 184.72.0.0/18;
deny 50.18.0.0/16;
deny 184.169.128.0/17;
deny 54.241.0.0/16;

# EU (Ireland)
deny 79.125.0.0/17;
deny 46.51.128.0/18;
deny 46.51.192.0/20;
deny 46.137.0.0/17;
deny 46.137.128.0/18;
deny 176.34.128.0/17;
deny 176.34.64.0/18;
deny 54.247.0.0/16;
deny 54.246.0.0/16;
deny 54.228.0.0/16;

# Asia Pacific (Singapore)
deny 175.41.128.0/18;
deny 122.248.192.0/18;
deny 46.137.192.0/18;
deny 46.51.216.0/21;
deny 54.251.0.0/16;

# Asia Pacific (Sydney)
deny 54.252.0.0/16;

# Asia Pacific (Tokyo)
deny 175.41.192.0/18;
deny 46.51.224.0/19;
deny 176.32.64.0/19;
deny 103.4.8.0/21;
deny 176.34.0.0/18;
deny 54.248.0.0/15;

# South America (Sao Paulo)
deny 177.71.128.0/17;
deny 54.232.0.0/16;

Now, this is accurate as of the time of posting, but don't assume this IP address range list to be static, you'll need to monitor the forum post I linked above!

Important Note

After doing any major change to your webserver like this, ALWAYS carefully monitor your webstats, Google Webmaster Tools, Bing Webmaster Tools and anything else which shows the continued health of your website. If it looks likely to be impacting your traffic and/or search positions badly, back it out immediately and consider if this is the cause.

Hope you had a nice Christmas, and have a happy new year. :)

Comments

Add Your Own Comment