《IBM Big Data Platform Overview.ppt》由会员分享，可在线阅读，更多相关《IBM Big Data Platform Overview.ppt（40页珍藏版）》请在三一办公上搜索。
1、IBM Big Data Platform Overview,Martin Pavlk+420 731 435 691martin_,Big Data is a Hot Topic Because Technology Makes it Possible to Analyze ALL Available Data,Cost effectively manage and analyzeall available data in its native formunstructured,structured,streaming,ERP,CRM,RFID,Website,Network Switche
2、s,Social Media,Billing,BIG DATA is not just HADOOP,Manage&store huge volume of any data,Hadoop File SystemMapReduce,Manage streaming data,Stream Computing,Analyze unstructured data,Text Analytics Engine,Data Warehousing,Structure and control data,Integrate and govern all data sources,Integration,Dat
3、a Quality,Security,Lifecycle Management,MDM,Understand and navigate federated big data sources,Federated Discovery and Navigation,Business-Centric Big Data Enables You to Start With a Critical Business Pain and Expand the Foundation for Future Requirements,“Big data”isnt just a technologyits a busin
4、ess strategy for capitalizing on information resourcesGetting started is crucial Success at each entry point is accelerated by products within the Big Data platformBuild the foundation for future requirements by expanding further into the big data platform,1 Unlock Big Data,2 Analyze Raw Data,Mergin
5、g the Traditional and Big Data Approaches,ITStructures the data to answer that question,ITDelivers a platform to enable creative discovery,Business Explores what questions could be asked,Business UsersDetermine what question to ask,Monthly sales reportsProfitability analysisCustomer surveys,Brand se
6、ntimentProduct strategyMaximum asset utilization,Big Data ApproachIterative&Exploratory Analysis,Traditional ApproachStructured&Repeatable Analysis,InfoSphere BigInsights is more than just HADOOP,Hadoop,Open-source software framework from ApacheInspired byGoogle MapReduceGFS(Google File System),HDFS
7、Map/Reduce,InfoSphere BigInsights,Platform for volume,variety,velocity Enhanced Hadoop foundation Analytics Text analytics&toolingApplication acceleratorsUsabilityWeb console Spreadsheet-style tool Ready-made“apps”Enterprise ClassStorage,security,cluster management IntegrationConnectivity to Netezza
8、,DB2,JDBC databases,etc,ApacheHadoop,Basic Edition,Enterprise Edition,Licensed,Application accelerators Pre-built applicationsText analytics Spreadsheet-style toolRDBMS,warehouse connectivity Administrative tools,securityEclipse development toolsPerformance enhancements.,Free download Integrated ins
9、tallOnline InfoCenterBigData Univ.,Breadth of capabilities,Enterprise class,Can run also on top of,Spreadsheet-style Analysis,Web-based analysis and visualization Spreadsheet-like interface Define and manage long running data collection jobsAnalyze content of the text on the pages that have been ret
10、rieved,Build a Big Data Program MapReduce example,Eclipse tools For Jaql,Hive,Pig Java MapReduce,BigSheets plug-ins,text analytics,etc.,JAQL IBMs programming language in hadoop world,Jaql is a complete solutions environment supporting all other BigInsights components,Integration point for various an
11、alyticsText analyticsStatistical analysisMachine learningAd-hoc analysis Integration point for various data sourcesLocal and distributed file systemsNoSQL data basesContent repositoriesRelational sources(Warehouses,operational data bases),BigInsights Text Analytics,Statistical Analysis(R module),Mac
12、hine learning(SystemML),Ad-Hoc analysis(BigSheets),(Integration)DB2,Netezza,Streams,Jaql,Jaql I/O,Jaql Core Operators,Jaql Modules,DFS,NoSQL,RDBMS,File System,BigInsights,Data warehouse,Traditional analytic tools,Big Data analytic applications,Filter Transform Aggregate,BigInsights and the data ware
13、house,3 Simplify your warehouse,Analyst,IT,I need to evaluate the possible relationship between client salary and overdrafts,OK.We have to evaluate a lot of statistics,set the correct db indexes and db partitioning.It will take us 5 days.,Analyst,IT,Great.Thanks a lot.Im going to check the results.,
14、Done.You can run your analytical query.,After 5 days.,Analyst,IT,Great.I can see here some nice correlations.Now I need to look at it from the different perspective.,After 10 minutes.,Ohhh,welcome dear friend.Understand.So,its.another 5 days of our work,Noooo!Its not possible to work here!,And now w
15、ith Netezza.,Analyst,IT,I need to evaluate the possible relationship between client salary and overdrafts.I will use Netezza.,Analyst,IT,Great.I can see here some nice correlations.Now I need to look at it from the different perspective.With Netezza I can run the query immediately.The response will
16、be in the same time,After 12 minutes.,IT can do something else much more useful,Go to View Header and Footer to change this footer text to the event title,23,Built-In Expertise Makes This as Simple as an Appliance,Dedicated deviceOptimized for purposeComplete solutionFast installationVery easy opera
17、tionStandard interfacesLow cost,IBM Netezza was renamed to IBM PureData System for Analytics,In October 2012,Netezza Genesis in T-Mobile CZ,Proof-Of-Concept ProjectNew EnterpriseDataWarehouse platform selectionComparison of existing and other platformsSelection CriteriaPerformanceOperational Savings
18、.and the winner was:Netezza,Netezza Genesis in T-Mobile CZ,ExpectationsSignificant response improvement:Faster platform means better reports responseDirect Data AvailabilityHigher trust in data,one version of truthAggregation reductionAny attribute availableOperational BenefitsStorage savings(no dat
19、a replicas)Administration costs reduction(DBA)Infrastructure SimplificationLower environment complexity,Netezza Genesis in T-Mobile CZ,Project ImplementationEDW platform migrationNetezza platform implementationETL graphs/processes redesignBI Front-End Tool MigrationSAP Business Object implementation
20、All reports redesignMain Integration Partner:T-System CZ,Netezza Genesis in T-Mobile CZ,Actual StatusAll relevant ETL procecessing redesignedActual parallel run to Original and Netezza platform finishedNetezza as only primary platform,RESPONSE TIME MASSIVELY IMPROVED,Real Netezza experience from T-M
21、obile Czech Rep.,4 Reduce costs with Hadoop,BigInsights and the data warehouse,BigInsights,Query-ready archive for“cold”warehouse data,Data Warehouse,Big Data analytic applications,Traditional analytic tools,From Cognos BIvia Hive JDBC,BigInsights Connectivity to DBMS/Warehouse,Netezza,BigInsights,J
22、DBCDBMS,DB2LUW,IW with DPF,BigInsights drives RDBMS work DBMS drives BigInsights work,Cognos Business Intelligence Big Data Architecture,Load through UDF,Netezza,RDBMS,Cognos Insight,Application(Map-Reduce,Lucene,SystemT),Storage(HBase,HDFS,GPFS),Query Methods(Jaql,Pig,Hive),CSV,InfoSphere BigInsigh
23、ts,REST via HTTP,Cognos BI Server,Text Analytics,REST API,Explore&Analyze,Report&Act,Visit the BigInsights wiki for a link to an article on Cognos and BigInsights,Application,SQL interface Engine,InfoSphere BigInsights,HiveTables,HBase tables,CSV Files,Data Sources,Future:The SQL interface.,Rich SQL
24、 query capabilitiesSQL 92 and 2011 featuresCorrelated subqueriesWindowed aggregatesSQL access to all data stored in InfoSphere BigInsightsRobust JDBC/ODBC supportTake advantage of key features of each data sourceLeverage MapReduce parallelismORachieving low-latency,5 Analyze Streaming Data,Why and w
25、hen to use InfoSphere Streams?,At least 2 criteria from the list bellow should be fulfilled,Applications needing on-fly processing,filtering and analyzing streaming data,Streams and Warehouse:Complementary,High,Med,Low,Low,Med,High,Latency,1PB,B,KB,GB,10GB,100GB,1TB,10TB,100TB,MB,Warehouse,Streams,S
26、calable processing of huge data stores,scalable low-latency processing of stream data,Unit of analysis,Warehouse,Streams,Warehouse/Hadoop,Streams,Sweet spot,Capa-bility,Sweet spot,Capa-bility,Streams and BigInsights-Integrated Analytics on Data in Motion&Data at Rest,1.Data Ingest,Data Integration,d
27、ata mining,machine learning,statistical modeling,Visualization of real-time and historical insights,3.Adaptive Analytics Model,Data ingest,preparation,online analysis,model validation,Data,2.Bootstrap/Enrich,Control flow,InfoSphereBigInsights,Database&Warehouse,InfoSphereStreams,The Platform Advantage,BI/Reporting,IBM Big Data Platform,IBM big data IBM big data IBM big data,IBM big data IBM big data IBM big data,IBM big data IBM big data,IBM big data IBM big data,THINK,
