Exclusive: Google, Amazon, and Microsoft Swarm China for Network Gear

About  10 years ago Cisco was the largest buyer of Flash memory. Is the recent demand for cloud storage, data centers, and servers a major source of demand for Flash and DRAM memories.

Ron Maltiel

Exclusive: Google, Amazon, and Microsoft Swarm China for Network Gear
http://www.wired.com/wiredenterprise/2012/03/google-microsoft-network-gear/
By Cade Metz Email Author March 30, 2012 | 6:36 am |

J.R. Rivers once built networking hardware at Google. Now he helps web
giants buy their networking hardware directly from China and Taiwan.
Photo: Jon Snyder/Wired
Google, Amazon, Microsoft, and Facebook buy more networking hardware
than practically anyone else on earth. After all, these are the giants
of the internet. But at the same time, they're buying less and less
gear from Cisco, HP, Juniper, and the rest of the world's largest
networking vendors. It's an irony that could lead to a major shift in
the worldwide hardware market.
Over the past few years, the giants of the web have changed the way
they purchase tens of thousands of the network switches inside the
massive data centers driving their online services, quietly moving
away from U.S.-based sellers to buy cheaper gear in bulk straight from
China and Taiwan. According to J.R. Rivers — an ex-Google engineer —
Google has built its own gear in tandem with varous Asian
manufacturers for several years, and according to James Liao — who
spent two years selling hardware for Taiwan-based manufacturer Quanta
— Facebook, Amazon, and Microsoft are purchasing at least some of
their networking switches from Asian firms as well.

"My biggest customers were these big data center [companies], so I
know all of them pretty well," Liao says. "They all have different
ways of solving their networking problems, but they have all moved
away from big networking companies like Cisco or Juniper or [the
Dell-owned] Force10."
The move away from U.S. network equipment stalwarts is one of the
best-kept secrets in Silicon Valley. Some web giants consider their
networking hardware strategy a competitive advantage that must be
hidden from rivals. Others just don't want to anger their business
partners in the hardware sector by talking about the shift. But cloud
computing is an arms race. The biggest web companies on earth are
competing to see who can deliver their services to the most people in
the shortest amount of time at the lowest cost. And the cheapest arms
come straight from Asia.

J.R. Rivers is one of the arms dealers. He runs a company called
Cumulus Networks that helps the giants of the web — and other outfits
— buy their networking hardware directly from "original design
manufacturers," or ODMs, in China and Taiwan. And he's worked in this
world for an awfully long time. He's one of the Google engineers who
secretly designed a new breed of networking switch for the company's
data centers, the massive computing facilities that drive its search
engine and the rest of its web services.
Rivers joined Google in October 2005, after five ears as a
distinguished engineer at Cisco, the company that dominated the
worldwide market for networking gear. At the time, Google was still
connecting its servers using standard networking switches from the
likes of Cisco and Force10 Networks. But these mass-market switches
just didn't suit Google's unusually large operation.
"When Google looked at their network, they need high-bandwidth
connections between their servers and they wanted to be able to manage
things — at scale," Rivers says. "With the traditional enterprise
networking vendors, they just couldn't get there. The cost was too
high, and the systems were too closed to be manageable on a network of
that size."
So Google drew up its own designs — working alongside manufacturers in
Taiwan and China — and cut the Ciscos and the Force10s out of the
equation. The Ciscos and the Force10s build their gear with many of
those same manufacturers. Google removed the middlemen.
The search giant does much the same with its servers, buying
custom-built machines straight from Asia rather than going through
traditional sellers such as Dell and HP. Because its web services were
used by such an enormous number of people, Google faced all sorts of
data center problems no one else faced — problems of power and space
as well as cost and logistics. So it built all sorts of custom
hardware to solve those problems.
"They all have different ways of solving their networking problems,
but they have all moved away from big networking companies like Cisco
or Juniper or Force10″
Now, the other giants of the web are running into the same issues, and
they too are going straight to Asia for hardware. Following closely
behind are companies that run large internal server farms, including
financial houses and healthcare outfits.
As J.R. Rivers serves this market with Cumulus Networks, James Liao is
doing much the same thing with a second startup called Pica8, offering
networking gear that comes straight from the ODMs. Pica8 is a spinoff
of Liao's former employer, Quanta — one of the companies that
manufactured Google's original networking switches, according to
Rivers.
According to Liao, tens of thousands of switches are already being
sold by the Asian ODMs directly to the likes of Amazon, Facebook, and
Microsoft. And that doesn't include the gear Google has bought over
the past seven years. "This is just the beginning," Liao says,
pointing out that these buyers operate the biggest data centers on
earth. These companies account for only a part of the
$7-billion-a-year Ethernet switch market, but as more and more outfits
move their operations into the proverbial cloud, the influence of
these web giants will only grow.
Liao estimates that Amazon, Microsoft, Facebook and others have bought
Asian network switches spanning "millions" of network ports — i.e.,
connections to servers — and he guesses that in 2011, about 60 percent
of these ports provided 10Gigabit Ethernet connections. According to
Matthias Machowinski — a directing analyst with Infonetics, a research
firm that tracks the networking market — the official market for
10Gigabit Ethernet spanned about 9 million ports in 2011.
J.R. Rivers declines to name the companies he's working with at
Cumulus Networks, but he confirms that some of the big-name web
outfits are already buying networking switches from ODMs in Asia. In
all likelihood, these companies are also purchasing switches from
other sources as well. Cisco says it has a "significant presence and
mindshare" in the big-name web market, and Juniper says it has a
relationship with all of the top five web players, pointing out that
data center networks require more gear than just switches. But the
market is on the move.
The Future of 'Web Giant 3.0′
"We are continuously exploring new infrastructure technologies that
may evolve further efficiencies across our portfolio. We normally have
discussions with ODMs and large and small OEMs to better understand
their capabilities and evaluate their products," reads a statement
sent to Wired by a Microsoft spokesperson and attributed to Dileep
Bhandarkar, a distinguished engineer who oversees the data centers
driving Microsoft's online services. But the statement did not
specifically address the purchase of networking gear.
Amazon did not respond to a request for comment about its hardware
practices, and a Google spokeswoman sent us a one-sentence statement:
"We work with a variety of vendors to manufacture the equipment we use
in our data centers," she said. These two companies — particularly
Google — are rather tightlipped about their data center practices.
"This supply chain change is nascent. But it's the most exciting thing
going on in Silicon Valley right now"
Facebook declined to discuss how it purchases networking gear, but in
response to secretive approach of Amazon and Google, the company has
openly discussed some of its other practices, and it has actually
shared its server and data center designs with the rest of the world.
It purchases its servers directly from Quanta and Wistron, another
Taiwanese ODM.
Martin Casado — the chief technology officier of a third Silicon
Valley networking startup, Nicira — confirms that the hardware market
is shifting to Asia. Offering a software platform that virtualizes
networking gear in much the same way that VMware virtualized servers,
Nicira helps some of the big web players build their networks. The
Nicira platform was designed specifically for companies along the
lines of Google that want to use cheap commodity switches to
physically construct their network but then do all the complex
management in software.
"If you're building web giant 3.0, you can go to Quanta in Taiwan and
buy crates … of switches," he says. "This supply chain change is
nascent. But it's the most exciting thing going on in Silicon Valley
right now."

Google Goes to Asia
According to J.R. Rivers, Google began work on its custom-built
networking switches in early 2005, before he arrived at the company.
In the beginning, River says, Google worked in tandem with Quanta and
other Asian ODMs. But eventually, he says, Google took all the
engineering work in house. Basically, he says, the company wasn't
happy with the work the ODMs did at the time. Google engineers would
design the switches, and then they would bring the completed designs
to contract manufacturers in Asia, outfits along the lines of Foxconn,
the Asian company that builds Apple's iPhones and iPads.
Google has never discussed its practices publicly, but rumors have
long indicated that the company built its networking switches in this
way. In 2007, research analyst Andrew Schmitt noticed that certain
manufacturers were producing enormous numbers of chips for 10Gigabit
Ethernet switches but that the switches themselves weren't actually
turning up on the market. "It didn't make sense to me why someone
would be building so much of a given component if there were no
customers that could use it," he says. "What I was able to determine
is that Google was purchasing switch chips straight from the comment
suppler."
"It didn't make sense to me why someone would be building so much of a
given component if there were no customers that could use it"
The switches Google was building typically sat at the top of a rack of
servers in the data center, connecting the servers to the rest of the
network. As Juniper points out, this is only part of the networking
hardware used in the data, but it's a large part.
Google, Rivers says, is a unique company. It has the wherewithal and
the talent to built its own switches, but other companies may not be
up to the task. With Cumulus Networks, J.R. Rivers and his partner,
Nolan Leake, are trying to grease the wheels. "[The other web players]
are trying to figure out what the best model is, and that's one of the
reasons we started up," Rivers says. "Google is unique in its
willingness to build something just because they know it can be done.
Most other people see a risk/reward trade-off. We seek to minimize
that risk."
Though Rivers declined to name the ODMs his company is working with,
he says that these are well-known manufacturers in Taiwan and China.
"We've been working for the last year on opening up a supply chain for
traditional ODMs who want to sell the hardware on the open market for
whoever wants to buy," he says. "For the buyers, there can be some
very meaningful cost savings. Companies like Cisco and Force10 are
just buying from these same ODMs and marking things up. Now, you can
go directly to the people who manufacture it."
This has become possible in recent years, Rivers says, because the
ODMs have slowly acquired more and more engineering talent. You can
now buy commodity gear from more places. "Networking is opening up
much like the transition from mainframes to RISC machines and later to
x86 servers," says Rivers' partner, Nolan Leake. "We're moving towards
a world where customers have more control over their destiny."
'The Arms Dealer'
Before spinning Pica8 out of Quanta, James Liao was already selling
similar networking switches to the big web players. Nicira's Martin
Casado refers to James Liao as "the arms dealer" in this networking
revolution. "He's the conduit between the rest of the world and
Quanta. He knows this space better than anyone," Casado says. "And I
love him because he talks like he's part of organized crime."
From July 2009 to September 2011, Liao was the senior director at
Quanta in charge of product strategy for network switching and data
center products. He was based in Silicon Valley, and his job was to
serve the giants of the web. He declines to go into much detail about
how these companies acquire their hardware, but he's unequivocal in
saying that the other big companies — Amazon, Microsoft, and Facebook
— are now following Google's lead in going directly to Asia for their
gear.

James Liao. Photo: Courtesy Liao
Networking switches, he says, have become a commodity. "They all use
the same chips. They have to same latency. They have the same
bandwidth. This is a clear signal that the hardware platform is
commoditizing," he says. "You can actually find a lot of [ODM]
suppliers that have the capability to manufacturer and design this
kind of platform."
Like Cumulus Network, Liao's new venture, Pica8, brings this low-cost
networking hardware to a much larger market. In the past, one of the
problems with buying directly from the ODMs is that you had build your
own software to drive your switches. But Pica8 aims to provide
software for those companies that don't want to build their own. The
company has open sourced an early version of this software — known as
Picos — and it plans to open source a more extensive version of the
platform next month.
"We give you the hardware and the software," Liao says. "If you take
our platform and compare it to Cisco, the protocol features we provide
and the hardware performance are all in the same range. The only
difference is that the price is 40 percent to 60 percent lower."
Though Pica8 spun off of Quanta, Liao says that the company will also
sell switches from other ODMs. But he declined to name them. But he
does say Pica8 is selling gear to Japanese telecom giant NTT and
Baidu, the company that dominates the Chinese search engine market.
Matthias Machowinski, of research firm Infonetics, says he is "very
much aware" of this trend, though he adds that it is extremely hard to
track. He says that the big web giants account for only a part of the
overall switch market — "the number of customers that might choose to
go down this route are very limited. Today, you can count them on one
hand, and maybe over the next two years, two hands might be enough" —
but he also acknowledges that as businesses move their applications
onto services such as Amazon EC2 and Microsoft Azure — rather than
running stuff in their own data centers — these web giants will
account for an even larger part of the switch market.

Like Server, Like Switch
This shadow networking market is a repeat of what happened in the
server world. Years ago, Google started building its own servers in
tandem with the Asian ODMs, and other web giants followed. These
companies are looking to save cost, but they're also looking to reduce
their power consumption, customizing machines so they're far more
efficient than their mass-market brethren.
In 2009, Google revealed some server designs it produced several years
before. But, as with networking practices, the company says very
little about its server gear. Amazon operates in much the same way.
But Facebook had taken a different approach. Last year, after building
its own data centers and working with various manufacturers to build
its own servers, the social networking giant open sourced these
designs to the rest of the world, hoping that others across the
industry can help improve those designs, buy more hardware based on
the designs, and ultimately drive down the price of the hardware.
"It's kind like buying couches. If you buy one, you go to a retail
store. If you buy 10,000 couches, you go straight to the factory"
This Open Compute Project already has several other big-name backers,
including Texas-based cloud computing outfit Rackspace and Japan's
NTT. And it doesn't stop at data centers and servers. Last month,
Frank Frankovsky — the ex-Dell man who oversees hardware design at
Facebook — told us that the company is in the midst of building its
own storage hardware and that these designs will be open sourced in
early May.
In these cases, Facebook and Amazon and Google and others bypassed
"original equipment manufacturers," or OEMs, such as Dell and HP. The
servers sold by the likes of HP and Dell are actually manufactured by
those same ODMs in Taiwan and China.
James Liao, of Pica8 and formerly of Quanta, does not work with
servers. But he says that it's common knowledge that — like Google and
Facebook — Amazon purchases at least some of its servers from ODMs in
Asia. "For servers, Facebook and Amazon are taking almost exactly the
same approach," he says. "Amazon also has some very high power
designers, but they don't do the design themselves. They come up with
a certain architecture and they tell the ODMs: 'This is my vision.
These are the goals. And I want help designing the hardware.'"
Now, Liao says, this same sort of thing is happening with, well,
everything. "All of the data center hardware is bought this way," Liao
says. "You can refer to Facebook as an example, where one of the big
projects inside the Open Compute effort is storage. Even the storage
side is being commoditized. Servers, storage, and networking — all of
them are going to this way."

Nolan Leake and J.R. Rivers of Cumulus Networks. Photo: Jon Snyder/Wired
Howard Wu — the president of greater China for Joyent, an Amazon-like
cloud computing outfit based in San Francisco — agrees. "If you're a
small business and you're going to buy five servers, you're going to
Dell or HP, because of the support services. But if you're a data
center operator and you're going to buy 10,000 servers, you're going
straight [to the ODMs]," he says. "It's kind like buying couches. If
you buy one, you go to a retail store. If you buy 10,000 couches, you
go straight to the factory."
That said, Joyent is not yet buying its gear from the ODMs.
"We are definitely in talks, but it hasn't actually happened yet," he
says. "We have other contractual obligations right now." The market
has not completely shifted to Asia. It's moving in stages. These web
companies have many suppliers — that's just good for businesses — and
in some cases, they're still buying hardware from the traditional
players — perhaps because they still have contracts in place.
Facebook, for instance, is still buying some servers from Dell and HP.
And Amazon is still buying custom servers from Rackable, a stateside
manufacturer, and apparently other outfits based here in America.
The hardware supply chain is vast and varied. But it's consolidating.
Now that they have the engineering talent, J.R. Rivers says, the ODMs
are transforming into OEMs. "The market is maturing to the point where
anyone can buy directly from ODMs," he says. "You don't have to be
Google."