How the cloud helped to deliver a smooth experience for the 2021 Australian Census

On the evening of 10 August 2021, millions of Australians took part in a national survey that helps to show how the economic, social, and cultural make-up of the country changes over time.

The Australian Bureau of Statistics (ABS) holds the Census of Population and Housing every five years. It is the most comprehensive snapshot of Australia, collecting data from around 10 million households and more than 25 million people. Governments and businesses use this data to make decisions about transport, schools, health care, roads, and buildings. It also helps them plan local services for individuals, families, and communities.

This year, the ABS chose to run the Census primarily online to create a smooth experience for people.

Building a scalable Census in the cloud

In 2016, the dedicated eCensus infrastructure experienced a series of denial-of-service (DDoS) attacks, a hardware router failure, and false positive report of data being exfiltrated. This caused ABS to temporarily shut down access to the 2016 Census website, which meant many Australians were unable to complete their 2016 Census forms in a timely manner.

For the 2021 Census, the ABS took a different approach and engaged PwC Australia (PwC) to create a robust Census Digital Service (CDS), hosted on Amazon Web Services (AWS), that would be able to handle the web traffic generated by millions of users. This is the first time the Census has been run on the cloud.

The PwC built solution on AWS was designed to enable ABS to automatically scale up the digital 2021 Census website as user demand increased, and deliver a fast and consistent experience for users regardless of how they completed their forms. In preparation for the 2021 Census, the service was thoroughly tested for extreme user demand at more than 2,000 times the expected peak workload, and was easily able to manage the 2.5 million people who submitted their forms on 2021 Census day. The CDS served over eight million forms over the nine-week period, and during its peak at 8:06pm AEST on 10 August, the ABS received about 142 online submissions per second and a peak of 249 logins per second.

In addition to supporting the digital delivery of the 2021 Census, ABS worked with AWS to build a cloud-based contact centre, which enabled Australians to automatically request paper forms at a time that suits them. People who called were prompted by an automated agent to enter a few details like their Census ID number and their postcode to be verified. This meant over 394,000 people did not have to call the staff in the Census Contact Centre to request a paper form, and saved thousands of hours of wait time.

Keeping the Census data secure

A critical requirement of the 2021 Census was to keep the service secure. AWS provided world-class infrastructure and security protections to support the ABS and PwC in their implementation of stringent security measures, to provide rigorous protection for the Australian Census data.

This included an independent security and compliance assessment against the Australian Government’s Information Security Manual, through an Information Security Registered Assessors Program (IRAP) assessment. All information collected in the digital 2021 Census service was securely stored in the AWS Sydney Region. It was also encrypted end-to-end, which means the information was scrambled and could not be read without the decryption keys, which were controlled solely by the ABS.

PwC and ABS used AWS services that had been assessed at PROTECTED, the highest level of security compliance possible for public cloud-based systems under the Australian government’s IRAP. These included tools that provide advanced detection and mitigation in the event of large and sophisticated DDoS attacks. A web gateway was developed to ensure each Census form was validated before being passed along to the ABS’s processing environment. ABS, AWS, and PwC also conducted extensive DDoS tests, which were equivalent to being in the top 1% of DDoS attacks observed by AWS globally.

The Census still attracted the attention of cyber criminals but the security measures put in place by AWS, PwC, and the ABS successfully blocked traffic from more than 130,000 malicious IP addresses during the Census period – a small number given the scale of the Census. On 11 August, the Australian Assistant Treasurer Michael Sukkar labelled the 2021 Australian Census a success, noting there were no interruptions, excessive wait times, or security issues.

Optimising online and offline experience

Australians are clearly showing a growing preference for digital interaction with government, and this is driving a growing demand for scalable, reliable, and secure online services. COVID-19 has generated a massive surge in online activity, with Wearesocial’s Digital 2021: Australia report finding the amount of time Australians users aged 16-64 years old are spending online increased by 10% in 2020, equal to an extra 32 more minutes each day.

AWS understands the demands of delivering essential government services online and has deep expertise in helping customers scale. In 2017, the ABS engaged AWS to set up a contact centre for the Australian Marriage Law Postal Survey. That project was delivered in less than 100 days and ran smoothly, demonstrating the agility and reliability of cloud-based solutions.

Another example is the Australian Electoral Commission (AEC), which transitioned its public websites to run on AWS to increase their flexibility and scalability, especially during elections when the websites experience heavy traffic. AEC expected its website to receive over 100 million hits on election night alone in 2019, and the AWS-powered platform scaled to support these heavy traffic spikes and peak capacity demands.

Building a digital future together

At AWS, we support the Australian Government’s vision for Australia becoming a leading digital economy and society by 2030, by supplying innovative and flexible cloud services and investing in training programs to ensure that the nation has a workforce that can fully harness the benefits of the cloud.

Through the AWS Partner Network, which includes many Australian-owned businesses, we are working closely with Australian public sector customers, including federal, state, and local government agencies to support their digital transformation efforts like the 2021 Census.

We are committed to working with the Australian Government to deliver future nation-building projects, and we look forward to continuing to help drive future innovation on behalf of our customers.