Amazon Web Services (AWS) is a giant in cloud computing. It provides a variety of services that facilitate managing and hosting infrastructure on the cloud. AWS is used widely by both companies and individuals to build or host any type of computer application. This includes, but not limited to, web servers, serverless applications, file and object storage, and much more.
AWS have become a go-to for both developers and businesses as they’re guaranteed a reliable service for all the computing needs. Although nowadays, there are many clouding hosting services that may provide similar features, it wasn’t the case before Amazon started building AWS.
Building AWS was a remarkable foresight from Amazon that understood what businesses and developers needed to continue innovating without limitations. This article explores the story of how Amazon built AWS.
Early Beginning of AWS
In the early 2000s, Amazon (named Amazon.com back then) was only focused on the retail industry. Although it had a software development team internally that built the Amazon store, it wasn’t considered a technology company and didn’t garner any interest from developers.
In 2002, Tim O’Reilly, a renowned author and founder of O’Reilly Media, visited Jeff Bezos to show him a tool he created called Amarank. This tool automatically opened Amazon’s store every few hours and scanned the ranking of O’Reilly Media books and its competitors’ books on Amazon’s ranking for books. It did this using screen scraping. This tool allowed O’Reilly to keep track of the rankings without having to manually check the Amazon store.
O’Reilly then suggested to Bezos that Amazon exposes its data, such as products or rankings, through APIs. This would allow other companies and developers to build powerful tools on top of Amazon.
“Companies need to think not just what they can get for themselves from new technologies, but how they can enable others” — Tim O’Reilly
After O’Reilly’s visit, Bezos discussed this issue with the top Amazon executives. It was then pointed out that an engineer named Rob Frederick, the founder of a mobile commerce startup called Convergence that Amazon acquired, was working on a similar project inside Amazon. He was creating APIs that would allow mobile devices to access Amazon’s store.
So, Bezos decided to add Frederick to an Associates group with the mission to create APIs that allow developers to build on top of Amazon’s store. These APIs would eventually allow developers to access and perform operations such as publishing products or using payments systems.
After building these APIs, Amazon held its first developer conference, making developers an essential asset to Amazon, similar to customers of the Amazon store. Then, the group that was run by Rob Frederick and the Associates group was named “Amazon Web Services”.
Amazon’s Struggle with Infrastructure
Around the same time Amazon was building the stepping stones of AWS, Amazon was facing troubles with their infrastructure. As their business was growing in traffic, and as they were adding more features into the Amazon store, their monolithic architecture built in the mid 90s was no longer sufficient.
So, Amazon decided to transition their infrastructure to a service-oriented architecture. This allowed them to rebuild their systems into independent but connected components. This transition took 3 years.
During this transition, Amazon teams struggled to access Amazon’s resources and infrastructure while building or testing new features. Although Amazon was moving into a more flexible architecture, its resources were still managed by a single team in the Seattle headquarters.
This problem delayed the roll-out of new features and frustrated both software developers at the company and the top executives, including Jeff Bezos. It was a recurring and visible problem, as teams would often present their new features to the executives, but admit they never got the chance to actually test it out.
Jeff Bezos then got inspired by a book he was reading at the time called Creation by Steve Grand. The book gave him the idea to build the essential building blocks of computing that would allow Amazon’s developers to freely explore, innovate, and be creative. This wouldn’t only solve Amazon’s infrastructure problems, but similar problems other companies and developers were evidently experiencing as well.
“Developers are alchemists and our job is to do everything we can to get them to do their alchemy” — Jeff Bezos
The first steps to building these building blocks was to write down the list of “primitives” of computing. Those included storage, bandwidth, payments, among other primitives. Then, Amazon started putting together teams to develop each of these primitives as services.
Building S3 and EC2
The first two services, which are essential services that developers use now, are Simple Storage Service, or S3, and Elastic Compute Cloud, or EC2.
The team assigned to build S3 worked closely with Jeff Bezos, who was highly interested in the development of these services. Bezos had big ideas and expectations of S3 that the team working on it struggled to keep up with.
While the team looked over details related to service growth and downtime, Bezos insisted that the service needs to be built from the start ready to scale infinitely and with no downtime.
On the other hand, EC2 was built in isolation by a team of two, Chris Pinkham and Chris Brown, working remotely from South Africa. The idea behind building EC2 was to allow developers to build and run their application on Amazon’s servers, regardless of what type of application it was. The plan was to build EC2 on top of an open source tool called Xen which made it possible to run several applications on one physical server.
The Launch of Amazon Web Services
S3 was launched in March 2006. It allowed developers and websites to store files such as photos and documents on Amazon’s servers. Although S3 is a popular service now, at the time it hardly got any traffic and was overlooked by developers and companies.
A few months later, EC2 was in public beta and available for developers to use. Developers rushed to get access to EC2 and start building their applications using Amazon’s servers. This ultimately lead to more traffic and usage of S3 as well.
These services were highly unanticipated by rival companies such as Google and Microsoft. Amazon was the only company dabbling with cloud computing and providing developers with services that allowed them to build with no limitations. As these services rose in popularity, rival companies started noticing what they were missing out on, which lead them to launch their own similar services (Microsoft announced Azure in 2010, and Google announced Compute Engine in 2012).
However, Amazon was already many steps ahead at that point. Other teams at Amazon were already working on and launching other services, such as Flexible Payment Services and Amazon CloudSearch.
The Impact of AWS
AWS transformed both Amazon and software development as we know it. Inside Amazon, teams started relying on AWS and testing it even when services were in early stages. This allowed them to find issues in AWS early on, and facilitated building and rolling out new feature. It solved a huge hurdle that the engineering teams within Amazon were facing.
Companies and startups also started relying heavily on AWS. It removed all restrictions they previously faced with building and creating applications regardless of the sector they worked in. AWS pioneered a new era powering innovation and creativity. Amazon was finally considered a technology company following their launch of AWS.
Amazon’s revolutionary work with AWS spawned from the need for flexibility within its rigid architecture at the time. It limited their innovation and ability to grow as an ecommerce business. Amazon realized something essential, and arguably ahead of its time: monolithic architecture hinders a business’s growth, and modularity facilitates it.
Amazon’s transition in their architecture and motivation behind building AWS was to provide developers, both within the company and globally, with the necessary tooling to build without limitations. As Bezos indicated at the time, trying to guess what a developer would need when building an application was a thing from the past.
Amazon's forward-thinking strategy of empowering developers through APIs and infrastructure services was a game changer for their cloud offering. As we look to the future of ecommerce, the role of developers in shaping the industry is critical.
While there is still progress to be made in optimizing developer experience, at Medusa we aim to follow in Amazon's footsteps by providing ecommerce developers with the necessary building blocks to build, scale, and innovate. With the right primitives in place, we believe that developers can build the Amazons of tomorrow.