Stevey's Google Platforms Rant: Awesome Insight Into the Web Platform Wars

FASCINATING blog post from a Google engineer who used to work for Amazon.com. While the gist is a rant about Google+, the bits I dig are around Amazon’s transformation into a Service Oriented Architectural company (as well as hearing some insights about Amazon, a company I greatly admire).

Some insights about massive SOA:

• “every single one of your peer teams suddenly becomes a potential DOS attacker. “
• “if you have hundreds of services, and your code MUST communicate with other groups' code via these services, then you won't be able to find any of them without a service-discovery mechanism. And you can't have that without a service registration mechanism, which itself is another service. So Amazon has a universal service registry where you can find out reflectively (programmatically) about every service, what its APIs are, and also whether it is currently up, and where.”
• “monitoring and QA are the same thing.” … “In order to tell whether the service is actually responding, you have to make individual calls. The problem continues recursively until your monitoring is doing comprehensive semantics checking of your entire range of services and data, at which point it's indistinguishable from automated QA. “
• “Organizing into services taught teams not to trust each other in most of the same ways they're not supposed to trust external developers.”
• “ Bezos realized long before the vast majority of Amazonians that Amazon needs to be a platform.”
• “That one last thing that Google doesn't do well is Platforms. We don't understand platforms. We don't "get" platforms.“
• “The Golden Rule of Platforms, "Eat Your Own Dogfood", can be rephrased as "Start with a Platform, and Then Use it for Everything."  "”

My whole love affair with SharePoint revolves around it being a platform. You can read the whole post here: https://plus.google.com/112678702228711889851/posts/eVeouesvaVX 

Hopefully they don’t yank it down!

Hosting Clockwork Web Framework With Amazon

I’ve blogged a lot about my admiration for Amazon’s web services stack. I think they understand the web as well as any company in the world. It’s always been my intention to investigate Amazon’s Electronic Compute Cloud (EC2) and since I needed hosting for my new Clockwork Web Framework, I decided to give it a try.

The reason I went with Amazon rather than a traditional hoster is that I have no idea what kind of interest there will be in the framework, and therefore cannot predict what the load on a web server will be. Amazon EC2 is designed for this kind of flexibility, and you pay per hour.

The Platform

I am running a small Windows Server 2003 32x server instance to begin with. It only has 1.7 gb of RAM. I can scale this up if I need to, or more likely I will run up another small instance and load balancing the two using Amazon’s Elastic Load Balancer technology.

On this, I am using IIS 6, .NET 3.5, SQL Server 2005 Express, and Powershell. Most of my files are kept on a permanent storage drive (more on this below) and served by IIS. In order to maximize the speed and lower the CPU burden on the server, I have decided to use another Amazon technology, CloudFront.

CloudFront Content Delivery Network

CloudFront is a Content Delivery Network (CDN), like Akamai or Limelight. I use it to serve my images and resource files. Basically Amazon has edge servers all over the world with a copy of my images and resource files, and when users request them from my website, CloudFront automatically sends them a copy from the nearest location to them, making for some very fast download times.

To make this work, you have to use Amazon Simple Storage System, or S3. This is a virtual file system. Basically you have “buckets” of files that are served up when requests come in from the CloudFront “distributions”.

I’ve optimized it a bit by having two distributions; one for images and one for resources. This means that a page which requires both things will load even faster since two parallel CDN distributions are processing the files at the same time.

You can create CloudFront distributions through code, or through Amazon’s web management portal.

  Since you can control the public URL of the distributions, you will notice if you view the properties of my website that my images are handled by the path “http://images.clockworkwf.com” and my resource files are handled by the path “http://resources.clockworkwf.com” . In other words, I have full control over what path I give them. Most people will never know these picture are being served from Amazon.

I notice the website loads really quickly, so the CloudFront makes a big difference.

EC2 Hosting Challenges

So that’s the high level architecture. There are a number of impacts when using Amazon as a hoster I’d like to talk about.

Server Goes Up, Server Goes Down

To begin with, you have to assume that at any moment your server will go down. If your server dies, it vanishes, and you have to “spin up” another one, using the web interface or code. It’s very easy to do from the web console, just click “Launch Instance” and you can pick any server ranging from Ubuntu Linux to Windows 2003 Server 64x Enterprise R2.

Although the server instances you can use have their own hard drive space on C: and D: drives, you have to treat that as transitory storage.

I’ve setup my system in such a way that I can use an Elastic Block Storage (EBS) hard drive volume, provided by Amazon.This is a more permanent drive space that you pay for, but can be attached to any server instance. Think of it as a SAN (that’s probably what it is).

So I’ve got my database and web files on this EBS block, which I then mount to any server instance I’m currently running.

On the server instance, I simply point IIS web server to the EBS block files, and away we go.

The EBS can be any size you like, and you pay per GB per month. Right now I’m using 10GB since my log files and database don’t take up much room. I can add more space later if I need to.

Here’s a screenshot of that EBS volume, in the Amazon web console.

Dynamic DNS Entries

Next problem: Since the server can go down at any moment, DNS is a problem. If my server dies and I spin another one up, it will be given its own IP address, which my DNS entry for www.clockworkwf.com wouldn’t know about. So there might be a long delay while DNS changes to the new IP address.

So, I’m using a Dynamic DNS service called Nettica. They have a management console where I can enter my various domain records and assign a short Time To Live (TTL), which means the DNS entries update frequently. So if my server dies, I can change the entry in Nettica to point to the new server’s IP address, and within a few seconds requests are going back to the right place.

Nettica even allows me to control all of this through C# code. Going forward I plan to write powershell server management scripts that can automatically spin up a new server on Amazon, determine the IP, and register that with Nettica.

Incidentally, Amazon EC2 allows you to buy what are called “Static IP Addresses”. Essentially you can “rent” a fixed IP address which can by dynamically allocated to a server instance. So, in the short run this makes life easier for me as I have rented one, used that for my Nettica domain name record, and can assign this fixed IP to any new server instance.

Next problem: Disaster Recovery.

Disaster Recovery is even more important in Amazon EC2 world than elsewhere, since again your instances could die at any moment….Not that they will, but the point is, they are “virtual” and Amazon isn’t making any promises (unless you buy a Service Level Agreement from them).

However, Amazon’s EC2 provides a level of DR by its very nature – you can spin up another machine in a small amount of time. Estimates for new Windows instances are about 20 minutes.

There’s also something called an Availability Zone. Essentially it means “Data Centre” – Amazon has several of these and so you can spread your servers around between US – East, US-West, Europe, and so on. So when that Dinosaur-killing comet hits North America, the Europe Availability Zone keeps chugging.

Right now I’m not really doing much with my database, so DR isn’t such an issue. I have some security since my files are on an EBS block. However, eventually I’ll setup a second server in another availability zone and load balance the two.

Another Challenge: Price

Amazon Web Services are flexible, and you are charged per hour, for only what you use. This is an amazing model but it doesn’t work so well for website hosting, because of course your servers are supposed to be online 24/7, 365 days a year.

It’s hard to tell for sure what the annual bill will be, but for my small server instance (remember, only 1.7 Gb of RAM) it will cost well over $1,000. That’s a lot more than shared space on a regular hosting provider. However I’m willing to pay this, for the flexibility I get, and also because I think Amazon web services are a strategic advantage and so the earlier I learn about them, the more business opportunities I might unlock.

One good thing is that Amazon has been aggressively dropping its prices as it improves its services. Additionally, they have started offering “Registered Servers” – basically a pre-pay option for 1-year, 2-year, and 3-year terms. Unfortunately these are only for Linux servers at the moment but hopefully they will add them for Windows and then I can save money year on year.

CloudHost Monitoring

Amazon offers a web-based monitoring option for its server instances. I’ve started using it (for an additional fee) but I’m not sold on its utility yet. I don’t think I’m using it to its full potential yet – it is supposed to help you manage server issues by monitoring thresholds.

Managing S3 Files Using Cloudberry Explorer

I needed an easy way to create and manage my buckets, CloudFront distributions, and S3 files. I found Cloudberry Explorer, and downloaded the free version of it. I was able to drag and drop 1600 files from my Software Development Kit to the S3 bucket where I’m serving the resources. Super!

There’s a pro version I might purchase which would allow me to set the gzip encryption and other properties on the files. This would help lower my bandwidth costs and speed up the transfer a bit.

Here’s a screenshot of Cloudberry in action:

I love how easy it is to setup and use Amazon’s web services stack. I think they have a great business model for the Cloud, and they’re the company to beat. I’m willing to rely on them for the launch of Clockwork Web framework and so far I haven’t been disappointed.

Above the Clouds: A Berkeley View of Cloud Computing

The Electrical Engineering and Computer Sciences department at the University of California at Berkeley just put out a research paper on Cloud Computing as they see it.

The paper is an in-depth exploration of what some consider to be just another buzzword, Cloud Computing. Since nobody has agreed on what exactly it means, the implication is that it's just a marketing term.

I remember when web services started to appear, around 2000/2001 if I recall correctly. The descriptions and possibilities seemed great, but nobody really knew what to do with them or why. So there came a time when nobody talked about web services anymore and it looked like that particular bubble had burst.

In fact, behind the scenes, a host of companies and individuals were figuring web services out, building their own, and releasing them. A couple of years after the term started popping up, web services arrived for real and now we have mashups and SaaS and Software + Services and some really well-traveled XML fragments zipping around the globe.

The same thing seems to be going on with Cloud Computing. We're in the early days, and still hearing the "Moon on a stick" promises that Cloud Computing is a silver bullet for everything.

This white paper is one of the first I've seen that really quantifies the (potential) cost savings of Cloud Computing.

Some gems:

• Explanations on Cloud Computing and how it differs from previous attempts;
• Classes of Utility Computing on page 10, comparing Google AppEngine, Amazon Web Services, and Microsoft's beta Azure platform;
• Cloud Computing economic models, on page 12;
• A discussion of the Top 10 challenges- and potential solutions to them - on page 16;
• The observation that FedExing your data is a good way to cut down on your bandwidth costs and delays.

This is very impressive work. The full paper is here:

http://www.eecs.berkeley.edu/Pubs/TechRpts/2009/EECS-2009-28.pdf

Amazon EC2 To Support Windows OS

Great news from the folks at Amazon Web Services - their EC2 Cloud computing platform will shortly support Windows operating systems. This is a wonderful development as it will allow .NET applications to be hosted with IIS at the same level of scalability as the Linux folks now enjoy. I've signed up so when the first beta comes out, I have a fighting chance to test it.

Pricing will be higher for Windows OS than for Linux of course but with the advances in virtualization these days I doubt it will be a huge difference. Of course what you are paying for is the extra services of (presumably) professional backup, redundant servers, locked down security, onsite engineers, and all the bells and whistles a massive data centre run by one of the world's largest e-tailers will provide.

I'm a keen fan of Amazon's work - I think of all the major players they are the ones who understand the new economics of the internet the best - yes even better than Google I would argue. They are commoditizing application development in a granular and sustainable way.

Of course Google's revenue stream is huge. Of course they have made major successes by providing services such as online office apps, email, maps, analytics, adsense and adwords, but at the end of the day their revenue is entirely search-based and it isn't clear what level of support  any of these additional services are likely to receive over time. Right now it doesn't matter that all of these services are "free" and "Beta", because they are intended to fuel Google's search revenue. However, that assumes that Google will remain the #1 search destination. If it doesn't, all of these services will have to be dropped or given some kind of business model. In my opinion, anyone building on these services is therefore taking a bit of a chance.

At the bottom of any software ecosystem, the founder is essentially bullet-proof. That's because anybody using the platform faces the switching cost to another ecosystem, and also because the more services are offered, the more compelling staying on the platform becomes.

What Amazon is quietly doing is encouraging everyone to build on their platform, but charging them for this. By doing so, they are making a sort of business guarantee - they get a revenue stream for each of their services, and in turn they can support it with Service Level Agreements and dedicated teams. In other words, there is a vision and a road map because Amazon's web services each raise money.

Recent improvements to Amazon's DevPay - which provides an easy way for developers to charge for their software - and their work on the super-scalable SimpleDB are proof that they are in this for the long haul.

At the end of the day, developing on any cloud computing platform  isn't just a technical challenge. It raises a lot of thorny questions - how do you protect your data, what kind of legal issues accrue, what are the privacy issues, how do you handle service level agreements, how do you pass on your costs - just to name a few. Cloud computing is not a silver bullet and anybody committing to it needs to take a deep breath before they do, and research and plan ahead.

However, the commoditization of computing resources represents the same impact as the introduction of the telephone line or automobile factory - after awhile you forget they are there and just RELY on them. Seen this way, a future timeline of technological progress might include the names, "Thomas Edison", "Henry Ford", and..."Jeff Bezos".

What do you think?

SiSense's Amazon S3 Analytics Tool: Update

Shortly after I posted about SiSense's new Prism Business Intelligence Viewer with Amazon S3 analytics add-on, their Product Manager Adi Azaria contacted me. I'd mentioned a bug that occurred when I was trying to view my files. He asked to see my Prism log files, which I sent, and provided me with the newest version of the product.

This version provides an enhanced UI including more help links, a set of "Common tasks", and the ability to quickly create new data connections. Most satisfying, it fixed my bug!

Looks like my bucket did not have logging turned on. The new version checks for this and prompts the user when this happens.

Thanks Adi for the quick follow-ups and the great support!

Amazon Simple Storage Service Analytics Add-On

ReadWriteWeb has a post on an Amazon Simple Storage Service S3 analytics add-on called Prism by the folks at SiSense.

It's in private beta right now but there is an invite code to the readers of the ReadWriteWeb blog. I've signed up to see how it can help me manage my Amazon S3 buckets.

Prism Viewer is a desktop installation that provides a Business Intelligence interface to view the S3 usage stats. The Amazon Analytics Dashboard is an additional file (Prism file: .psm) that lives inside the Prism Viewer and reports on your S3 buckets.

Once you've entered your Amazon S3 connection information, you can start to create reports on its activity.

Unfortunately I can't synchronize my bucket's logs because an error occurs. It is a private beta but I'm disappointed I can't yet see my own statistics. For the moment here's a screenshot of their sample bucket to indicate what the tool will look like:

I'll try again later and see how it works. Since I plan to store more and more information on S3 I definitely need this kind of reporting.

Amazon is starting to build a great ecosystem on top of its utility cloud computing platform, and software such as Prism, JungleDisk, TagCow, and DreamTeam Suite is the proof.