《智慧型家庭网路之技术与应用.ppt》由会员分享，可在线阅读，更多相关《智慧型家庭网路之技术与应用.ppt（51页珍藏版）》请在三一办公上搜索。
1、1,智慧型家庭網路之技術與應用,Professor Yau-Hwang KuoDirectorCenter for Research of E-life Digital Technology(CREDIT)National Cheng Kung UniversityTainan,Taiwan,Outline,IntroductionStructure of Smart Home NetworkRealization of Device&Network LayersAgent-based Platform Affective HCIIntegrated PerceptionCognition L
2、ayerSmart Home ServicesConclusion,Trend of Digital Home,House_n(MIT)、Aware Home(Geogria Tech.)、Interactive Workspace(Stanford Univ.)、MavHome(UTA)。Digital Home Working Group:HP,Intel,IBM,.ECHONET:Energy Conversation and Homecare Network.CELF:Consumer Electronic Linux Forum.OSGi:Open Service Gateway I
3、nitiative Easy Living:Microsoft,Scenarios of Digital Life,smart digital housekeeper.ubiquitous digital nursing agent.affective digital tutor.ubiquitous home security monitor.ubiquitous home content service.universal cyber circles.ubiquitous universal messaging service.personal knowledge warehouse/na
4、vigation.nomadic personal digital secretary.secure traffic navigator.,Microsofts View for Digital Home Solution,Total connectivityNo more islands of functionalityPersonalized experiencesCustomized entertainment,communications,and controlUbiquitous accessYour PCs,devices,and content,securely accessib
5、le everywhere,Microsofts View for Digital Home Solution,Technology“by invitation only”,not imposedHighly personal and personalized spaceVirtually random,unmanaged“build out”Complex mix of products and services,Issues of Digital Home,人機互動能否人性化？robustness、adaptability、multi-modal collaboration 人性化互動特質
6、。感官、認知、情緒、協調、合作實現人性化互動的技術要素。ubiquitous multi-modal affective human-machine interaction 數位家庭的人性化互動需求。,Issues of Digital Home(cont.),人際互動能否得到提昇擴大？去空間限制、去時間限制、去工具限制、去安全限制。家電間的協力合作能力能否得到提昇?connectivity among appliances、autonomous collaboration of appliances、interoperability of appliances。,Issues of Digi
7、tal Home(cont.),人在數位生活空間的自由度是否得到提昇？可移動性、可轉移性、可調整性。ubiquitous integration home network、location-awareness、universal access、multi-modal human-machine interaction,Issues of Digital Home(cont.),人在數位生活空間的便利度是否得到提昇？生活機能完整性、設備與網路無縫結合度、生活機能可獲性(availability)、用戶干預度、操作易度、穩私與安全等。人在數位生活空間所獲得的生活輔助機能能否得到提昇?Smart h
8、ome network is necessary!,Goals:Infrastructure&applications,Create a new life space supported by a smart home service network and attached digital appliances.Develop e-services over the smart home network and digital appliances to realize a new life style.Develop a service modeling and execution env
9、ironment over the smart home network to realize various e-services.,Goals:technologies,Develop nomadic HCI technologySpeech,vision,physiology,sensors.Develop affective HCI technologyDevelop agent-based home service network middleware.Develop embedded platform&SoC for smart appliances.,Applications(h
10、ealth care,entertainment,surveillance,etc.),Application Layer,Speech,Vision(face),Physiology,Smell,Perception Layer,Home Network(802.11,Bluetooth,HomePlug)+Mobile Internet(SIP+3G),Network Layer,Layered Structure of Smart Home Service Network,Service Model Execution Platform(script translation,schedu
11、ling,QoS),Integrated Perception,Device Layer,Networked Physiology&EnvironmentMonitoring Appliances,Networked Microphones;Cameras;Speakers,Home Comm.Gateway;Home Perception Server;Home Media Center,WirelessA/V StreamingAppliances,Mobile Agent Platform,Agent Layer,Emotion/Semantics/Behavior/Intention
12、Understanding,Corpus of Knowledge(Ontology),InferenceEngine,Natural LanguageProcessing(text,spoken),Cognition/Affection Layer,Vision(gesture),Device&Network Layers:types of digital appliances,Client-type devices802.11g-based multifunctional audio/voice adaptor802.11g/MPEG-4-based multifunctional vid
13、eo adaptor802.11g/MPEG-4-based smart IP cameraBluetooth-based ECG device Gateway-type devicesMultimedia communication gatewayServer-type devicesHouse control server Human-machine interaction serverContent serverApplication server,Device&Network Layers:relationship among server appliances,用戶端設備,FTTH/
14、3G/WiMAX,通訊伺服器,CO,Internet/WWW,Telephony,house control&housekeeping devices,data store,A/V devices,WiFi/Home Plug,WiFi/Home Plug,主人,應用伺服器,屋控伺服器,WiFi/Home Plug,WiFi/Home Plug,內容伺服器,Scheduling Algorithm,XML,Service Agent,Task agent,Task agent,Task agent,ASI,BIS,PMS,LKN,FEA,ASI_1_1,ASI_1_2,BIS_1_1,BIS_
15、2_1,PMS_1_2,PMS_1_1,LKN_1_1,LKN_2_1,FEA_1_1,FEA_2_1,Service Server,Register,SDHScenarioServer,DB,User Request,LocationServer,Script,XML,What to do?,How to do?,Where to do?,CommonAPI,Architecture of agent platform,Agent-based Runtime Environment,Execution environment:IBM Aglets systemCommon API,Adapt
16、ive Service Provider:architecture,Adaptive Service Provider:functionalities,FunctionalitiesRegistry mechanism for subsystem,device and functionalitiesService provider for user requestsLoad balanced service scheduling algorithm according to system resources Agent cooperation mechanism,Adaptive Servic
17、e Provider:components,Service server Subsystem and devices functionalities registration Service portal for usersMonitoring each subsystem and device Service agents Provide service for each user requestService composition Task assignment and task agent dispatch according to predefined XML-based scena
18、rios,Adaptive Service Provider:components(cont.),Task agent Execute each functionality on each subsystemCommon APIService scheduling algorithmProvide a task list for service agent according to registry and pre-defined scenarios in databaseA Petri net based&load balanced scheduling algorithm for adap
19、tive service path in each subsystem and device,Agent-based Middleware:mobility management,Location detectionDevice-followed type:mobile IP;signal analysisDevice-free type:speech interaction;vision monitoring.Seamless handoff and transcoding for ubiquitous service following Roaming path tracking and
20、prediction,Agent-based Middleware:appliance collaboration management,Collaboration among homogeneous appliances:data fusion,task migration.Collaboration among heterogeneous appliances:multi-modal HCI.Scheduling,concurrency control&synchronization of collaborative tasks.Self-organization for service
21、deployment,Agent-based Middleware:interoperability management,Device bridgeProtocol bridgeTranscryptionTranscodingContent translation&adaptation,Agent-based Middleware:remote access management,Remote service deploymentremote service accessremote service managementauto-configurationservice re-directi
22、onservice aggregationUI remoting,Agent-based Middleware:other management functions,load management:Client-server load partitionServer load sharing Load scheduling of appliance farm availability managementFault toleranceJust-in-time activation of appliances service quality management,Affective Speech
23、 Conversation,Emotional Speech Synthesis,Behavior Understanding by Vision,High-Level behavior understanding from videosState MachineHuman Activity RecognitionTwo-Stage recognition processAccident/Abnormal behavior detectionContext&domain knowledge Combination,System Architecture,Method Activity Reco
24、gnition,Activity RecognitionLevel 1-posturesPosture SequenceLevel 2 motion/historyHistory Map Matching,Method Behavior understanding,BehaviorNormal behaviorState MachineActivity+ContextsAbnormal behaviorNormal behavior+domain knowledgeAccidentUnreasonable activity+domain knowledge,Facial Expression
25、Analysis,Face Acquisition,Acquisition,Segmentation,Facial Feature Extraction,Deformation Extraction,Motion Extraction,Representation,Facial ExpressionClassification,Recognition,Key frameSelection,YCbCrColor space,Eye Region,Mouth Region,RegionOfInterest,EyePoints,MouthPoints,Displacement Vectors,Fuz
26、zyNeuralNetwork,InvariantMoments,OpticalFlow,Key Frame,Image Sequence,Integrated Perception:fuzzification of reference perceptual models,Manipulate all kinds of perception in a uniform process to ease the perceptual integration.Due to high vagueness of perception,fuzzy logic based approach is a good
27、 choice to establish the reference models of perception.The reference models which fuzzify perceptual attributes and perceptual decision subspaces will be embedded into the integrated perception model.,FL-based Acoustic Reference Model for Emotion Recognition,feature extraction,AAU1 model,SVM cluste
28、ringfor emotion 1,SVM clusteringfor emotion 2,SVM clusteringfor emotion V,speech corpus,AAU2 model,AAUS model,fuzzification of acoustic features(AFs)and construction of acoustic action units(AAUs),FL-based Acoustic Reference Model for Emotion Recognition(cont.),Adopt SVM clustering approach in the s
29、ubspace of each emotion type to gather the clusters of acoustic training patterns.Inspect all produced SVM clusters in the whole feature space and merge the highly overlapped clusters.Each cluster is modeled as an AAU represented with its fuzzy cluster center where each feature is a fuzzy set whose
30、membership function is determined by the least-square curve fitting approach on the feature values of training samples included in the cluster.,FL-based Acoustic Reference Model for Emotion Recognition(cont.),The mapping between AAUs and emotion types is dependent on the SVM clustering result of eac
31、h emotion type.Each emotion type is associated with a set of clusters of acoustic samples.The weight of each cluster is determined by the ratio of the number of samples it contains with respect to the total amount of samples of the same emotion.,FL-based Facial Reference Model for Emotion Recognitio
32、n,graphical head model,morphological process to simulate AUs,FACS AUs identification process,feature points(FPs)extraction process,fuzzy logic based reference model for FACS,correspondence,Membership grade,Membership grade,FP1 value,FP2 value,FAU1,FAU2,FAUi,FAUj,FAUk,FL-based Facial Reference Model
33、for Emotion Recognition(cont.),Intend to construct a computational reference model for FACS action units based on the measurable features of facial expression.An approach similar to the construction of acoustic reference model is adopted.The training samples are generated from a generic head model w
34、ith necessary morphological manipulation.,FL-based Facial Reference Model for Emotion Recognition(cont.),The membership functions will be determined by the least-square curve fitting approach according to the sample patterns produced from the morphological process.Each AU may just represent a partia
35、l facial expression and relate to more than one emotion.,FAU1,FAU2,FAUK,AAU1,AAU2,AAUS,FP1,FPn,AF1,AFm,emotion type layer,representative concept layer,scaled feature layer,primaryfeature layer,Face Features Expression,Acoustic Features,Fuzzy Neural Network for Integrated Emotion Recognition,Fear,Ang
36、er,Surprise,Fear,Anger,Surprise,Fuzzy group decision process,group level of agreement,Fuzzy Neural Network for Integrated Emotion Recognition(cont.),All kinds of perceptual information are fused by the FNN model to realize emotion recognition.Each appliance will have an instance of the corresponding
37、 FNN to join the emotion recognition job.A two-layered(emotion type&concept layers)BP learning algorithm is adopted by using the training samples in constructing reference models.The fuzzy group decision process does not join the learning.Scaling input value to 0,1 in the second layer is realized by
38、 the membership function of the corresponding fuzzy set.,Fuzzy Neural Network for Integrated Emotion Recognition(cont.),The links between AUs and scaled features are not fully connected.The FAU/AAU nodes realize normalized weighted sum for the membership grades of input features weighted by their re
39、spective link strength.Each emotion type node determines output value by the normalized weighted sum of its inputs from the representative concept layer.,Cognition Layer:understanding and response,Understand the semantics of multi-modal expression.Classify and recognize the intention/need/emotion of
40、 semantic expression.Summarize the semantics of multi-modal expression according to classified result.,Cognition Layer:understanding and response(cont.),Predict the user behavior sequence according to the classified result.Schedule the response sequence according to the prediction result.Determine t
41、he instant response.,Stimulus,Perception,Cognition,spokenlanguage,gesture,faceexpression,physiologicalsignals,text,SpeechProcessing,VisionProcessing,SignalProcessing,VideoProcessing,SpeechProcessing,ApplicationControl,Response,SemanticFeatureExtraction,Conceptualization,Event Detector(Neural Network
42、-based Approach),EmotionRecognition,EmotionEpisodeDiscovery,Ontology,PersonalEvent/Emotion Log,ContextualRules,StimulusSemanticSummaryExtraction,User BehaviorPrediction(Episode-based Approach),ResponseScheduling,InstantResponseDetermination,StimulusResponseTemplates,EventSequenceCase base,EmotionSeq
43、uenceCase base,SemanticExpression,EmotionAttributes,Features,Concepts,Events,SemanticSummary,Emotion Episodes,PredictionResult,Response,Emotion Types,ResponseRoadmap,Smart Home Services,nomadic content serviceshealth care by integrated perceptionsmart home surveillancesmart e-mail and calendar arran
44、gement,Conclusion,Life style of human being will be heavily affected by ICT,but the technological gap is still big.Ubiquitous HCI and OCI technologies will be important to realize digital life style.Cognitive computing and affective computing are important to improve the effectiveness of HCI technol
45、ogy.,Description of Context-Aware Middleware,UserProfile,AdmissionControl,PersonalAgent,ContextReasoning,ContextAggregator,ResourceManagement,Wrapper,Wrapper,Service,Device,ServiceAgent,Context-Aware Middleware Architecture,OSGi Platform,UPnPBundle,ContextReasoningBundle,ResourceManagementBundle,WrapperBundle,LocationDetection,SpeechRecognition,PostureRecognition,Identification,Context Aggregator and Ontology Reasoning,Interface,Bundle Repository,JADE Agent Platform(Service Scenario),JAVA Virtual Machine,Operation System,
