Dmytro Shypovalov sent me his views on the hardware differences between routers and switches. Enjoy!
So, a long time ago routers were L3 with CPU forwarding and switches were L2 with ASIC. Then they had invented TCAM and L3 switches, and since then ASICs have evolved to support more features (QoS, encapsulations etc) and store more routes, while CPU-based architectures have evolved to specialised NPU and parallel processing (e.g. Cisco QFX) to handle more traffic, while supporting all features of CPU forwarding.
At some point, the 2 approaches intersect which allows vendors to market a pipeline-based device (with ASIC and TCAM) as a router. A classic example is Cisco rebranding the 6500 switch as the 7600 router. The 6500 with the proper supervisor (3BXL if memory serves me) already could handle BGP full view at the time, also it supported all MPLS features except VPLS. It could even fit some WAN linecards with non-ethernet ports. But it was still marketed as a switch! In the 7600 they added more WAN linecards with metro ethernet, advanced QoS almost like on software routers (shaping, hierarchical policies etc) and VPLS. They did that by offloading that advanced logic to new linecards, so whether your features would work depended on what is the ingress and the egress linecard for the given traffic flow. I supported 7600 when I worked in Cisco TAC, and half the cases started from figuring out to which linecards the source and destination are connected and then looking at a big spreadsheet which would tell whether this setup is supposed to work.
Another problem with pipeline architectures is that if a certain feature can’t be handled by ASIC, packets will be punted to CPU. Apart from potentially causing a huge performance problem, it can also break features (QoS, ACL etc) because the packet is taken out of the pipeline for CPU handling. This is an endless rabbit hole. How about sending first TCP SYN packet to CPU for MSS clamping (and then let the CPU do ECMP hashing), but let ASIC do ECMP hashing for other packets of the same flow, if we are using anycast and want all packets of the flow to be sent out of the same interface? This doesn’t occur (or occurs much less) in “True routers” with CPU-based architectures. To make it worse, it is often very unobvious whether the packet will be handled by CPU or ASIC. You can configure exactly the same thing in slightly different ways and get different results (e.g. on our beloved 6500/7600, PBR with set interface vs set ip nexthop, or empty sequence PBR – I crashed a border router once this way!). Or many GRE tunnels terminated on the same IP, the list goes on and on. And of course recirculation, which not only halves performance but can also break some features.
When Cisco announced EoL of 6500 and 7600, they advised that customers replace 6500 with Nexus 7000 (switch with a switch) and 7600 with ASR9000 (router with a router). N7k and ASR9k are fundamentally different platforms with different architecture and different OS, also the latter is much more expensive. So lots of people thought that they can use N7k where they used 6500 or 7600 – for example, to run MPLS services or RSVP-TE. This was a very bad idea. The only time in my life I saw a routing loop in a single-area link-state IGP (which in theory should not be possible) was on Cisco Nexus. And the MPLS implementation there was even worse. Anyway, this is a rant about software quality, not platform architecture per se.
As Mark Twain famously said – “History doesn’t repeat itself, but it often rhymes.” So 20 years after the 6500/7600, we got Arista 7280/7500R series – initially marketed as switches, but with development of an advanced routing suite in EOS, the next R2 and R3 models are marketed as routers. Indeed, they can handle BGP full view and support most MPLS and other advanced routing features, but they don’t support all of those simultaneously. So if you want to run MPLS-EVPN, BGP full view, advanced QoS, BGP flowspec and VXLAN routing on the same box – good luck with that. And then there are countless small annoying bugs or hardware limitations related to encapsulations – GRE (MTU enforcement, TTL handling), MPLS (max labels to push, explicit null handling, ECMP hashing), UCMP and the list goes on and on. All of those things sort of work, until they don’t and then the customer is not happy to hear that a box they bought as a router is, in fact, a fancy switch.
To conclude, what is the difference between routers and switches in my opinion? I have absolutely no idea. All vendors use merchant silicon now, so they will run into similar limitations, but might try to work around them in different ways. I think the discussion about platform architecture (hardware pipeline vs software forwarding) can be more productive.