《google文件系统.ppt》由会员分享，可在线阅读，更多相关《google文件系统.ppt（24页珍藏版）》请在三一办公上搜索。
1、GFS:The Google File System,Brad KarpUCL Computer Science,CS Z03/403030th October,2006,2,Motivating Application:Google,Crawl the whole webStore it all on“one big disk”Process users searches on“one big CPU”More storage,CPU required than one PC can offerCustom parallel supercomputer:expensive(so much s
2、o not really available today),3,Cluster of PCs as Supercomputer,Lots of cheap PCs,each with disk and CPUHigh aggregate storage capacitySpread search processing across many CPUsHow to share data among PCs?Ivy:shared virtual memoryFine-grained,relatively strong consistency at load/store levelFault tol
3、erance?NFS:share fs from one server,many clientsGoal:mimic original UNIX local fs semanticsCompromise:close-to-open consistency(performance)Fault tolerance?,4,Google Platform Characteristics,100s to 1000s of PCs in clusterCheap,commodity parts in PCsMany modes of failure for each PC:App bugs,OS bugs
4、Human errorDisk failure,memory failure,net failure,power supply failureConnector failureMonitoring,fault tolerance,auto-recovery essential,5,Google File System:Design Criteria,Detect,tolerate,recover from failures automaticallyLarge files,=100 MB in sizeLarge,streaming reads(=1 MB in size)Read onceL
5、arge,sequential writes that appendWrite onceConcurrent appends by multiple clients(e.g.,producer-consumer queues)Want atomicity for appends without synchronization overhead among clients,6,GFS:Architecture,One master server(state replicated on backups)Many chunk servers(100s 1000s)Spread across rack
6、s;intra-rack b/w greater than inter-rackChunk:64 MB portion of file,identified by 64-bit,globally unique IDMany clients accessing same and different files stored on same cluster,7,GFS:Architecture(2),8,Master Server,Holds all metadata:Namespace(directory hierarchy)Access control information(per-file
7、)Mapping from files to chunksCurrent locations of chunks(chunkservers)Manages chunk leases to chunkserversGarbage collects orphaned chunksMigrates chunks between chunkservers,9,Chunkserver,Stores 64 MB file chunks on local disk using standard Linux filesystem,each with version number and checksumRea
8、d/write requests specify chunk handle and byte rangeChunks replicated on configurable number of chunkservers(default:3)No caching of file data(beyond standard Linux buffer cache),10,Client,Issues control(metadata)requests to master serverIssues data requests directly to chunkserversCaches metadataDo
9、es no caching of dataNo consistency difficulties among clientsStreaming reads(read once)and append writes(write once)dont benefit much from caching at client,11,Client API,Is GFS a filesystem in traditional sense?Implemented in kernel,under vnode layer?Mimics UNIX semantics?No;a library apps can lin
10、k in for storage accessAPI:open,delete,read,write(as expected)snapshot:quickly create copy of fileappend:at least once,possibly with gaps and/or inconsistencies among clients,12,Client Read,Client sends master:read(file name,chunk index)Masters reply:chunk ID,chunk version number,locations of replic
11、asClient sends“closest”chunkserver w/replica:read(chunk ID,byte range)“Closest”determined by IP address on simple rack-based network topologyChunkserver replies with data,13,Client Write,Some chunkserver is primary for each chunkMaster grants lease to primary(typically for 60 sec.)Leases renewed usi
12、ng periodic heartbeat messages between master and chunkserversClient asks server for primary and secondary replicas for each chunkClient sends data to replicas in daisy chainPipelined:each replica forwards as it receivesTakes advantage of full-duplex Ethernet links,14,Client Write(2),All replicas ac
13、knowledge data write to clientClient sends write request to primaryPrimary assigns serial number to write request,providing orderingPrimary forwards write request with same serial number to secondariesSecondaries all reply to primary after completing writePrimary replies to client,15,Client Write(3)
14、,16,GFS:Consistency Model,Changes to namespace(i.e.,metadata)are atomicDone by single master server!Master uses log to define global total order of namespace-changing operationsData changes more complicatedConsistent:file region all clients see as same,regardless of replicas they read fromDefined:af
15、ter data mutation,file region that is consistent,and all clients see that entire mutation,17,GFS:Data Mutation Consistency,Record append completes at least once,at offset of GFS choosingApps must cope with Record Append semantics,18,Applications andRecord Append Semantics,Applications should include
16、 checksums in records they write using Record AppendReader can identify padding/record fragments using checksumsIf application cannot tolerate duplicated records,should include unique ID in recordReader can use unique IDs to filter duplicates,19,Logging at Master,Master has all metadata informationL
17、ose it,and youve lost the filesystem!Master logs all client requests to disk sequentiallyReplicates log entries to remote backup serversOnly replies to client after log entries safe on disk on self and backups!,20,Chunk Leases and Version Numbers,If no outstanding lease when client requests write,ma
18、ster grants new oneChunks have version numbersStored on disk at master and chunkserversEach time master grants new lease,increments version,informs all replicasMaster can revoke leasese.g.,when client requests rename or snapshot of file,21,What If the Master Reboots?,Replays log from diskRecovers na
19、mespace(directory)informationRecovers file-to-chunk-ID mappingAsks chunkservers which chunks they holdRecovers chunk-ID-to-chunkserver mappingIf chunk server has older chunk,its staleChunk server down at lease renewalIf chunk server has newer chunk,adopt its version numberMaster may have failed whil
20、e granting lease,22,What if Chunkserver Fails?,Master notices missing heartbeatsMaster decrements count of replicas for all chunks on dead chunkserverMaster re-replicates chunks missing replicas in backgroundHighest priority for chunks missing greatest number of replicas,23,File Deletion,When client
21、 deletes file:Master records deletion in its logFile renamed to hidden name including deletion timestampMaster scans file namespace in background:Removes files with such names if deleted for longer than 3 days(configurable)In-memory metadata erasedMaster scans chunk namespace in background:Removes u
22、nreferenced chunks from chunkservers,24,GFS:Summary,Success:used actively by Google to support search service and other applicationsAvailability and recoverability on cheap hardwareHigh throughput by decoupling control and dataSupports massive data sets and concurrent appendsSemantics not transparent to appsMust verify file contents to avoid inconsistent regions,repeated appends(at-least-once semantics)Performance not good for all appsAssumes read-once,write-once workload(no client caching!),
