Performing from Behind the Great Firewall of China

The opportunity to do business in China presents a financial windfall for any company, just ask Apple (News - Alert), which last quarter reported profits were up 31 percent on strong iPhone sales in China. But there’s an obstacle to doing business in China that every multi-national enterprise faces: the Great Firewall of China. The Chinese government has implemented a system that blocks websites and content it considers to be unfavorable leaving the company via the Internet.

The byproduct is severe disruptions to the speed, reliability and performance of Internet-dependent business systems. For example, Cedexis’ China ISP Page Load Time report for Nov 23. 2015 showed 7 to 40 percent errors, 59-92 percent availability and 2.2 to 5.5 second median transaction times for synthetic transaction tests.  In comparison, the Cedexis US report shows 1 to 5 percent errors and over 95 percent available. 

This is a problem for global enterprises that use a combination of internal, cloud operated (AWS/Azure) and SaaS (News - Alert)-based applications.  For example, I recently worked with a global consulting firm that couldn’t certify their Chinese consultants because they couldn’t complete web-based training – which was hosted in the U.S. by a SaaS provider.  This not only creates technical issues for IT, but business and liability issues for the organization. 

The key to improving performance is to implement a simple three-step improvement process: Understand, Optimize and Control.

The Great Wall you cannot see from the space station

The Great Firewall is part of the broader Golden Shield Project, a censorship and surveillance project launched in 2003. The Great Firewall system is distributed across China, so there are several Internet peering points.  Each peering point has a suite of security systems that process every connection to and from the Internet for China.  This includes DNS redirection, firewalls IDS/IPS, content proxies and more.  The challenge we’re faced with is peering points often have scaling issues, performance problems or block business traffic.  This in turn impacts business velocity within the region. 

You might think a simple solution would be for an enterprise to place servers closer geographically to mainland China, such as in Hong Kong.  However, any Internet-hosted server that remains outside mainland China’s firewall is subject to the roadblocks and speed bumps the firewall creates.

Moving infrastructure into China helps, although it is not always completely reliable. Many web applications have dependencies to external services.  Due to this, businesses will still experience performance problems if its sites use third-party services not hosted inside of the firewall. For example, U.S.-based third-party services, like those provided by Microsoft (News - Alert) O365 or SalesForce, can cause a site that would otherwise load cleanly to load slowly or hang.

The key to identifying and implementing a system that will work best for your enterprise’s unique needs is to follow a three-step process: understand, optimize and control.

Understand

This sounds obvious, but it’s an important point to make: With end-to-end understanding comes critical insight into a system.  A complete understanding of the constraints the Great Firewall places on your operations will enable you to implement the fixes that overcome the firewall’s constraints. You need end-to-end real-time insight into performance issues related to transient performance issues.  This includes not just network telemetry, such as TCP (retransmissions, latency, reordering), or detailed multi-segment analysis (e.g., where was the packet lost/delayed/reordered), but also browser level application telemetry (DNS time, time to first byte, HTTP errors, etc.) as well as deep application performance telemetry (dependency maps, external calls, response times by function/method, errors, etc.). 

If possible, distribute synthetics test agents both in mainland China sites and in outside regions to monitor specific application transaction health and performance.  This can include not just internal applications, but SaaS or Cloud applications as well.  When performance issues arise, you will be able to more confidently triage issues, or track them over time for repeatable patterns.

What is critical, is you have complete end-to-end visibility into not just utilization, but more importantly response times and errors so you isolate key constraints (or identify new ones, should you move the application).

Optimize

After you gain an understanding of application performance issues, you can take steps to optimize your systems to overcome key constraints – such as the latency and errors caused by the Great Firewall.  The actions you can take will vary depending on the type of application.  Here are a few common optimization steps most organizations can take, prior to moving an application into a region (which is much more costly). 

1. If you own the web application, implement web optimization best practices per Google PageSpeed or Yahoo YSlow recommendations.
2. Use HTTP caching to store web objects closer to the users. For example in WAN Optimizers at remote sites or in open-source web cache in a regional peering point in China.  Make sure your approach can support HTTPS and is trusted as part of your Public Key Infrastructure (PKI) architecture.
3. Use WAN optimization to speed up business transactions 2 – 33x by reducing application and network turns through the Great Firewall, and across last-mile circuits to users.  This can also be very useful for SaaS hosted applications, such as Microsoft O365, Salesforce, Box and others where you may not own the infrastructure.

Keep in mind, the fastest round trip time is the one you don’t have to make, and the fewer you make through the Great Firewall the better.

Control

After achieving full visibility and using that information to intelligently optimize your systems, you can create more predictability by applying traffic engineering to address performance along delivery paths. Of course, you cannot directly control how the Great Firewall constrains traffic but you may be able to minimize the impact, or go around it altogether.

For example, if a site has multiple paths, say the Internet (through the Great Firewall) vs. MPLS (which backhauls the Internet or connects to a Carrier Neutral Facility), you can steer important business applications down the best performing path to prevent failures or poor performance.  In SD-WAN solutions, building business-centric policies to intelligently direct traffic down cheap (sometimes poor performing) Internet vs. expensive high-performing MPLS is easier than ever.

Often the last-mile WAN link is still a key constraint, so don’t forget to leverage application-layer Quality of Service for a consistent experience.  If possible, leverage integrated performance visibility in your edge devices to create a closed loop continuous improvement process. 

In some cases, you won’t be able to find a way to sneak under, over or around the Great Firewall of China, you simply have to work within the constraints it applies to your systems and applications. In addition to working to understand key constraints and optimize your systems to improve performance and give you more control, keep in mind that China’s web infrastructure is still being built and improved.