1、7/9/2001,Edward Chow Content Switch 1,Introduction to Linux-based Virtual Server and Content Switch C. Edward Chow Department of Computer Science University of Colorado at Colorado Springs chowcs.uccs.edu The ppt file of this tutorial is available at http:/cs.uccs.edu/chow/pub/conf/pdcat/tutorial.pp
2、t,Part of this work sponsored by CCL/ITRI,7/9/2001,Edward Chow Content Switch 2,Outline of the Talk,Overview of Content Delivery Networks Linux-based Virtual Server Linux-based Content Switching,7/9/2001,Edward Chow Content Switch 3,Clients,Content Delivery Network (CDN),Host Server,Mind Spring,PSIN
3、et,Home,Huge Requests,Server Crash,Slow Response,Clients,Clients,7/9/2001,Edward Chow Content Switch 4,Content Delivery Problems,http:/,7/9/2001,Edward Chow Content Switch 5,Use Client Cache/ Client Side Cache Server,Host Server,Mind Spring,PSINet,Sprint,Gloobix,Home,UUnet,Fewer Requests,Clients,Cli
4、ents,Clients,Client Cache,Client Side Cache Server,Fast Response,7/9/2001,Edward Chow Content Switch 6,Use Mirror Sites,Host Server,Mind Spring,PSINet,Home,Fast Response,Clients,Clients,Clients,Mirror Site,Mirror Site,Need improvement by guiding the selection of mirror servers with server load/netwo
5、rk bandwidth measurement,7/9/2001,Edward Chow Content Switch 7,Edge Network Cache Servers,Host Server,Mind Spring,PSINet,Home,Fast Response,Clients,Clients,Clients,Client Cache,Mirror Site,Mirror Site,Edge Network Cache Server,Cache Server,Cache Server,Cache Server,Cache Server,Client Side Cache Ser
6、ver,7/9/2001,Edward Chow Content Switch 8,Content Delivery Problem,Cache Location Problem: Where to put cache servers? How many are needed? When/where/how to push/delivery the content? How about dynamic content?,7/9/2001,Edward Chow Content Switch 9,Akamai Edge Delivery Service,Peering Bottleneck Pr
7、oblem: Access traffic evenly spread over 7400+ networks (no one over 5%; most 1%) Need to put edge servers in many networks. 11/2000, 4 billion bits/day for 2800 sites. Source Http:/,7/9/2001,Edward Chow Content Switch 10,Caching Dynamic Content at Web Proxies,Active Cache Project : PeiCao 98 Univ.
8、Wisconsin Cache Java applet to be executed at proxies Choice of passing to server, delivery cached copy, or generate dynamically. Edge Side Include (ESI): XML tag to specify ESI fragment in a web page. Each ESI fragment can have different cache/,7/9/2001,Edward Chow Content Switch 11,Edge Side Inclu
9、de Example http:/www.esi.org/, click here - - ,7/9/2001,Edward Chow Content Switch 12,Solution to First Mile Problem,First Mile Problem: Hugh requests at web site of CDN High Bandwidth Connection Caching End System Cache Client Cache Client Site Proxy Cache Server Mirror Site Caches Cache Servers in
10、 Internet Hierarchical Cache Servers, e.g., Squid/Harvest/Adaptive Web Edge Servers of Akamai Faster Server/Server Farm (Server Side Caching+Cluster) Layer4 Load balancer+Real Servers Content Switch+Real Servers Distributed Packet Rewrite,7/9/2001,Edward Chow Content Switch 13,Load Balancer or Conte
11、nt Switch,Real Server,Web Server Cluster,Load balancer can run atApplication Level Reverse ProxyKernel level Linux Virtual Server,Load balancer can distribute requests based on Layer 3-4 info fixe field/fast hash Layer 3-7 info var. length/slow parsing,Real Server,Real Server,Real Server,7/9/2001,Ed
12、ward Chow Content Switch 14,Comparison of Load Balancers,Reverse Proxy runs as application process requires more memory/packet copying. Linux Virtual Server runs in kernelno memory,7/9/2001,Edward Chow Content Switch 15,Linux Virtual Server (LVS),“Virtual server is a highly scalable and highly avail
13、able server built on a cluster of real servers. The architecture of the cluster is transparent to end users, and the users see only a single virtual server” with Virtual IP address (VIP). Http:/www.linuxvirtualserver.org/,Internet,VIP,Load Balancer/Director Linux Box,WAN/ LAN,Real Server1,Real Serve
14、r2,Real Server3,RIP1,RIP2,RIP3,CIP,Client,CIP: Client IP Address VIP: Virutal IP Address RIP: Real Server IP Address,7/9/2001,Edward Chow Content Switch 16,LVS-NAT Configuration (Network Address Translation),All return traffic go through DirectorSlow Modify IP addr/port #/Checksum at Director Direct
15、or and real servers at same LAN No modification needed on real-servers Port remapping: real web server can run on 8080,Internet,VIP,Director,Real Server1,Real Server2,Real Server3,RIP1,RIP2,RIP3,CIP,Client,Switch,7/9/2001,Edward Chow Content Switch 17,LVS-NAT Configuration Step 2. Director routes Pk
16、t,Based on CIP, source port#, VIP and dst port#, director selects one of the real servers Change the dst IP addr or port # of pkt.,Internet,VIP,Director,Real Server1,Real Server2,Real Server3,RIP1,RIP2,RIP3,1. request,2. Scheduling/ Rewrite packet,CIP,Client,Switch,LVS Routing Scheduling Rules,ipvsa
17、dm cmd,7/9/2001,Edward Chow Content Switch 18,LVS-NAT Configuration Step 3. Real Server Replies,Real server retrieves response. All real servers set default gateway to Director; like any other NAT or IP masquerade setup Packet will be sent back to Director.,Internet,VIP,Director,Real Server1,Real Se
18、rver2,Real Server3,RIP1,RIP2,RIP3,1. request,2. Scheduling/ Rewrite packet,CIP,3. Process Request,Client,Switch,RIP1 CIP,7/9/2001,Edward Chow Content Switch 19,LVS-NAT Configuration Step 4. Director rewrites reply,Director changes the dst IP addr. (RIP1) of pkt to VIP Modify port # if needed. Modify
19、 the checksum; send back pkt.,Internet,VIP,Director,Real Server1,Real Server2,Real Server3,RIP1,RIP2,RIP3,1. request,2. Scheduling/ Rewrite packet,CIP,3. Process Request,4. Rewrite reply,Client,Switch,RIP1 CIP,7/9/2001,Edward Chow Content Switch 20,LVS-NAT Configuration (Network Address Translation)
20、,All return traffic go through DirectorSlow Modify IP addr/port #/Checksum at Director. Director and real servers at same LAN,Internet,VIP,Director,Real Server1,Real Server2,Real Server3,RIP1,RIP2,RIP3,1. request,2. Scheduling/ Rewrite packet,CIP,3. Process Request,4. Rewrite reply,5. Receive reply,
21、Client,Switch,RIP1 CIP,7/9/2001,Edward Chow Content Switch 21,LVS-NAT Setup Commands,# make the director forward the masquerading packets echo 1 /proc/sys/net/ipv4/ip_forward ipchains -A forward -j MASQ -s 172.16.0.0/24 -d 0.0.0.0/0 # Add virtual service and link a scheduler to it ipvsadm -A -t 202.
22、103.106.5:80 -s wlc (Weighted Least-Connection scheduling) ipvsadm -A -t 202.103.106.5:21 -s wrr (Weighted Round Robin scheduling ) #Add real servers and select forwarding method and weight ipvsadm -a -t 202.103.106.5:80 -R 172.16.0.2:80 -m ipvsadm -a -t 202.103.106.5:80 -R 172.16.0.3:8000 -m -w 2 i
23、pvsadm -a -t 202.103.106.5:21 -R 172.16.0.2:21 -m,7/9/2001,Edward Chow Content Switch 22,LVS-Tunnel Configuration (IP Tunneling),Real Servers need to handle IP over IP packets. Real Servers can be geographically separated and return traffic go through different routes. Security implication!,Internet
24、,VIP,Load Balancer Linux Box,Real Server1,Real Server2,Real Server3,RIP1,RIP2,1. request,2. Scheduling/ Put packet in IP Tunnel,CIP,3. Process Request,4. Receive reply,Client,RIP0 RIP2,IP Tunnel,IP Tunnel,IP Tunnel,RIP3,RIP0,7/9/2001,Edward Chow Content Switch 23,LVS-Tunnel Setup Commands,#The load
25、balancer (LinuxDirector), kernel 2.2.14echo 1 /proc/sys/net/ipv4/ip_forward ipvsadm -A -t 172.26.20.110:23 -s wlc ipvsadm -a -t 172.26.20.110:23 -r 172.26.20.112 -i #The real server 1, kernel 2.2.14echo 1 /proc/sys/net/ipv4/ip_forward # insert it if it is compiled as module insmod ipip ifconfig tunl
26、0 172.26.20.110 netmask 255.255.255.255 broadcast 172.26.20.110 up route add -host 172.26.20.110 dev tunl0 echo 1 /proc/sys/net/ipv4/conf/all/hidden echo 1 /proc/sys/net/ipv4/conf/tunl0/hidden,7/9/2001,Edward Chow Content Switch 24,LVS-DR Configuration (Direct Routing),Real servers need to configure
27、 a non-arp alias interface with virtual IP address and that interface must share same physical segment with load balancer. Only Directors interface replies to VIP ARP request. Director only rewrites server MAC address; IP packet not changed Fast!,Internet,VMAC,Director,Real Server1,Real Server2,Real
28、 Server3,RMAC1,RMAC2,RMAC3,1. request,2. Scheduling/ Rewrite packet,CIP,Client,Route/Switch,GMAC: Gateway MAC address,7/9/2001,Edward Chow Content Switch 25,LVS-DR Configuration Step 3. Process Request,Real server returns request. Request goes directly through switch/router; not Director.,Internet,V
29、MAC,Linux Director,Real Server1,Real Server2,Real Server3,RMAC1,RMAC2,RMAC3,1. request,2. Scheduling/ Rewrite packet,CIP,3. Process Request,4. Receive reply,Client,Switch,GMAC: Gateway MAC address,7/9/2001,Edward Chow Content Switch 26,LVS-DR Configuration (Direct Routing),Real servers need to confi
30、gure a non-arp alias interface with virtual IP address and that interface must share same physical segment with load balancer. Load balancer only rewrites server MAC address; IP packet not changed Fast!,Internet,VMAC,Linux Director,Real Server1,Real Server2,Real Server3,RMAC1,RMAC2,RMAC3,1. request,
31、2. Scheduling/ Rewrite packet,CIP,3. Process Request,4. Receive reply,Client,Switch,GMAC: Gateway MAC address,7/9/2001,Edward Chow Content Switch 27,LVS-DR Setup Commands,#The load balancer (LinuxDirector), kernel 2.2.14 or later echo 1 /proc/sys/net/ipv4/ip_forward ipvsadm -A -t 172.26.20.110:23 -s
32、 wlc ipvsadm -a -t 172.26.20.110:23 -r 172.26.20.112 g #The real server 1, 172.26.20.112, kernel 2.2.14 or laterecho 1 /proc/sys/net/ipv4/ip_forward ifconfig lo:0 172.26.20.110 netmask 255.255.255.255 broadcast 172.26.20.110 up route add -host 172.26.20.110 dev lo:0 echo 1 /proc/sys/net/ipv4/conf/al
33、l/hidden echo 1 /proc/sys/net/ipv4/conf/lo/hidden,7/9/2001,Edward Chow Content Switch 28,Persistence Handling in LVS,Sticky connections Examples: FTP control (port21), data (port20) For passive FTP, the server tells the clients the port that it listens to, the client initiates the data connection co
34、nnecting to that port. For the LVS/TUN and the LVS/DR, LinuxDirector is only on the client-to-server half connection, so it is imposssible for LinuxDirector to get the port from the packet that goes to the client directly. SSL Session: port 443 for secure Web servers and port 465 for secure mail ser
35、ver, key for connection must be chosen/exchanged. Persistent port solution: First accesses the service, LinuxDirector create a template between the given client and the selected server, then create an entry for the connection in the hash table. The template expires in a configurable time, and the te
36、mplate wont expire until all its connections expire. The connections for any port from the client will send to the server before the template expires. The timeout of persistent templates can be configured by users, and the default is 300 seconds,7/9/2001,Edward Chow Content Switch 29,HA-LVS Configur
37、ation High Available,Linux Director,Real Server1,Real Server2,Real Server3,CIP,Client,Heart Beat,Backup Director,1. When Backup Director detects Linux Director failure through heart beat protocol, “graciously negotiate” the take-over of VIP Provide fault-tolerant,2. Monitor server processesrun on re
38、al servers Route requests to server processes that are alive. Initiate restart/repair,7/9/2001,Edward Chow Content Switch 30,Performance of LVS-based Systems,“We ran a very simple LVS-DR arrangement with one PII-400 (2.2.14 kernel)directing about 20,000 HTTP requests/second to a bank of about 20 Web
39、 servers answering with tiny identical dummy responses for a few minutes. Worked just fine.” Jerry Glomph Black, Director, Internet & Technical Operations, RealNetworks“I had basically (1024) four class-Cs of virtual servers which were loadbalanced through a LinuxDirector (two, actually - I used red
40、undant directors) onto four real servers which each had the four different class-Cs aliased on them.” “Ted Pavlic“ ,7/9/2001,Edward Chow Content Switch 31,LVS Usage Survey 2/15/2001 Lorn Key,C. Edward Chow Department of Computer Science University of Colorado at Colorado SpringsSponsored by Computer
41、 Comm. Lab/ITRI,7/9/2001,Edward Chow Content Switch 33,Content Switch Topics,What is a Content Switch? What Services it Can Provide Content Switch Example Related Technologies Content Switch Architecture and Basic Operations TCP Delay Binding and Related Improvement Content Switch Rule and Conflict
42、Detection Conclusion,7/9/2001,Edward Chow Content Switch 34,Content Switch (CS),Route packets based on high layer (Layer 5/7) headers and content. Examples: Direct Web traffic based on pattern of URLs, cookies URL Switching XML Tag Value Web Switching Can Route incoming email based on email address;
43、 Connect POP/IMAP based on login Web switches and Intel XML Director/accelerator are special cases of content switch.,7/9/2001,Edward Chow Content Switch 35,What Services It Can Provide,Enabling premium services for e-commerce, ISP, and Web hosting providers Load Balancing and High Available Server
44、Clusters: Web, E-commerce, Email, Computing, File, SAN Policy-based networking, differential/QoS services. Firewall, Strengthening DoS protection, cache/firewall load-balancing Flash-crowd management Email Spam Protection, Virus Detection/Removal Applet Authentication/Filtering,7/9/2001,Edward Chow
45、Content Switch 36,F5 VRM Solution,7/9/2001,Edward Chow Content Switch 37,Intel Netstructure XML Director 7280,Example of Rule: Server1: create */order.asp & /AmountValue = 10000,7/9/2001,Edward Chow Content Switch 38,Phobos In-Switch,Only load balancing switch in a PCI card form factor Plugs directl
46、y into any server PCI slot Supports up to 8,192 servers, ensuring availability and maximum performance Six different algorithms are available for optimum performance: Round Robin, Weighted Percentage, Least Connections, Fastest Response Time, Adaptive and Fixed. Provides failover to other servers fo
47、r high-availability of the web site U.S. Retail $1995.00,7/9/2001,Edward Chow Content Switch 39,E-Commerce Example: 1. Client,Client submits via HTTP/Post (or SOAP) the following purchase in XML:CCL111222333309121544IBM Thinkpad T2150001050000309121538Intel wireless LAN PC Card20010200052000 ,7/9/2001,Edward Chow Content Switch 40,
