Android多媒体框架概述.pdf

Android Multimedia Framework Overview Li Li, Solution and Service Wind River | 2010 Wind River. All Rights Reserved.2 Agenda What is Multimedia in a mobile device MPEG standard File at Codec Android Multimedia Framework OpenCORE OpenMAX What we can do with this | 2010 Wind River. All Rights Reserved.3 What is Multimedia in mobile device Multimedia is the most important component in modern mobile device with modem Multimedia framework is used to process video/audio and output to satisfy certain functionality including Video and Audio Storage video and audio in media Playback video and audio Record video and audio Audio, Video devices/ Perfect Audio Video | 2010 Wind River. All Rights Reserved.4 What is their relationship Container at specify how to wrap various meta-data/stream, most is coded data by codec Theoretically, a container at could wrap any kinds of data, most container ats are specialized for specific data requirements Container does not describe how the data warped is encoded. Always a program be able to identify and open a file ,but not be able to decode contained data. You maybe were told to download right decoder. Codec/Decode and File at | 2010 Wind River. All Rights Reserved.5 Codec/Decode device or computer program capable of encoding and/or decoding a digital data stream or signal A codec encodes a data stream or signal for transmission, storage or encryption and decode it for playback or editing. codec coder decoder Raw multimedia data is huge, codec compress them to facilitate store and transfer Codec/Decode and File at | 2010 Wind River. All Rights Reserved.6 File at Also call container or wrapper at Specify how different data elements and metadata coexist in a computer file or stream Always contain coded video, coded audio, subtitles, chapter- ination, maybe advertisement and synchronization ination needed to playback various streams together Codec/Decode and File at | 2010 Wind River. All Rights Reserved.7 Codec - Audio AAC, MPEG-4 Audio Part 3 subpart 4 AC-3, Dolby Digital codec AMR, Adaptive Multi-Rate Audio codec AMR-WB, Adaptive Multi-Rate Wideband MP2, MPEG1/2 Audio Layer II MP3, MPEG2 Audio Layer III Vorbis, OGG audio, opensource project Wma, Windows Media Audio RealAudio, RealNetwork ALAC, Apple Lossless Audio Codec Codec/Decode and File at | 2010 Wind River. All Rights Reserved.8 Codec -Video MPEG 1, MPEG 1 Part 2 MPEG 2/H.262, MPEG 2 part 2 MPEG 4 ASP, MPEG 4 Part 2 MPEG 4 AVC/H.264, MPEG 4 Part 10 VC 1 The inal name of SMTP 421M video codec standard Initially developed as a propriety video at by Microsoft before it was released as a al SMPTE standard video at on April 3, 2006 VC 2 An open and royalty-free video compression at, Dirac. 2010 the SMPTE standardized Dirac Pro as VC 2. VC 3 DNXHD, Digital Nonlinear Extensible High Definition a lossy high-definition video post-production codec engineered for multi-generation compositing with reduced storage and bandwith requirements The DNXHD codec was ted to the SMPTE organization as the framework for the VC-3 family of standard. Codec/Decode and File at | 2010 Wind River. All Rights Reserved.9 Standard MPEG Moving Picture Experts Group. A working group of ISO/IEC in charge of the development of international standards for compression, decompression, processing and coded representation of moving pictures audio and their combination The standard for storage and retri of moving pictures and audio on storage media. Approved Nov. 1992 VCD MP2, MPEG-1 Audio Layer II The standard for digital television. Approved Nov. 1994 DVD MP3, MPEG-2 Audio Layer III Codec/Decode and File at | 2010 Wind River. All Rights Reserved.10 MPEG -4 The standard for multimedia application a developing standard divided into a number of parts Most are left to individual developers to decided whether to implement probably no complete implementations of the entrire MPEG-4 set of standards To deal with this, the standard includes concept of “profiles” and “levels” allowing a specific set of capabilities to be defined in a manner appropriate for a subset of applications Codec/Decode and File at | 2010 Wind River. All Rights Reserved.11 MPEG-4 part 2, Advanced Simple Profile used by codecs such as DiVX, Xvid, Nero Digital and 3ivx, Quicktime 6 MPEG-4 part 10, MPEG-4 AVC Advanced Video Coding/H.264 used by x264 encoder, Nero Digital AVC, Quicktime 7, and high- definition video media like Blu-ray Disc Codec/Decode and File at | 2010 Wind River. All Rights Reserved.12 Video/Audio container MPEG program stream standard container for MPEG-1 and MPEG-2 elementary streams on reasonably reliable media such as disks; used also on DVD- Video discs MPEG-2 program stream, MPEG-TS Standard container for digital broadcasting and for transportation over unreliable media; used also on Blu-ray Disc Vedio; typically contains multiple video and audio stream, and an electronic program guide MP4 Standard audio and video container for the MPEG-4 multimedia portfolio, based on MPEG-4 Part 12 and JPEG 2000 Part 12 Codec/Decode and File at | 2010 Wind River. All Rights Reserved.13 Container at parser to recognize and unwrap file Codec to encode/decode data. Synchronization among various stream Memory/Buffer management Stream track control, playback, backwards play, forward play Integrated into video/audio output system Take advantage of hardware accleration Hardware codec Hardware overlay Hardware audio flinger What need to do with Multimedia | 2010 Wind River. All Rights Reserved.14 Android Multimedia Framework | 2010 Wind River. All Rights Reserved.15 OpenCORE Initial Android Multimedia Framework Android multimedia subsystem provided by PacketVideo modular, extensible framework Combining independent media processing components file ats, codecs, streaming protocol components rendering compoments other elements in different ways to implement a wide variety of multimedia scenarios abstraction a conception of NODE the base of the modulization Implementation in of NODE for recognizer module, parser module, codec module, sink module | 2010 Wind River. All Rights Reserved.16 OpenCORE Initial Android Multimedia Framework | 2010 Wind River. All Rights Reserved.17 OpenCORE Initial Android Multimedia Framework Paser Node File Handler Linux hardware codec driver VDEC NodeSink NodeVMIO ADEC NodeSink NodeAMIO Sync OMX HW CodecOMX SW Codec Data Control | 2010 Wind River. All Rights Reserved.18 OpenCORE Initial Android Multimedia Framework TypeCodecDecoderEncoderComments VideoH.263yesyesBaseline Profile MPEG-4yesyesSimple Profile AVC/H.264yesyesConstrained Baseline Profile AudioMP3yesno AACyesnoAAC, HE-AAC v1/v2 AMR-NByesyes AMR-WByesno Container atParserComposer MPEG4yesyes 3GPPyesyes 3GPP2yesno MP3yesno AACyesno AMRyesyes WAVyesno | 2010 Wind River. All Rights Reserved.19 OpenMAX Bridge Codec/Decode to Multimedia Framework the Integration Layer IL interface with Open Core royalty-free, cross-plat API for comprehensive streaming media codec and application portability enable accelerated multimedia components to be developed, integrated and programmed across multiple operating systems and silicon plat The OpenMAX API will be shipped with processors to enable library and codec Rapidly and effectively make use of the full acceleration potential of new silicon regardless of the underlying hardware architecture | 2010 Wind River. All Rights Reserved.20 OpenMAX - Framework | 2010 Wind River. All Rights Reserved.21 OpenMAX Position in Android Multimedia Framework OMX core Software Decoder Component H.263 Decoder Component H.264 Decoder Component AAC Decoder Component H263 Decoder Component H263 Decoder Component H263 OMX core Hardware OpenCore | 2010 Wind River. All Rights Reserved.22 OpenMAX API list For each OMX core, there are function points need to be implmented. pOMX_Init pOMX_Deinit pOMX_ComponentNameEnum pOMX_GetHandle pOMX_FreeHandle pOMX_GetComponentsOfRole pOMX_GetRoleofComponent pOMX_SetupTunel pOMX_GetContentPipe pOMXConfigParser | 2010 Wind River. All Rights Reserved.23 One Sample of Openmax Design | 2010 Wind River. All Rights Reserved.24 Functionality Does you support enough container at Does you support necessary codec Does you support stream, video over air Perance, critical Video perance. Frame per second Audio perance. High quality sound Synchronization between video and audio Commercialization | 2010 Wind River. All Rights Reserved.25 Take advantage of SoC capability Design OMX IL layer for specific hardware codec component Design Overlay take advantage of hardware gpu or video output controller Design high quality audio solution Whole system video buffer management scheme Share memory buffer between various hardware component Decrease memory copy as possible Synchronization scheme. Esp for some fault file Choose the right point to fix sync problem. In parser or codec. Commercialization | 2010 Wind River. All Rights Reserved.26 Surface FlingerVideo output Overlay DataGralloc V4L2 driverFramebuffer Commercialization GPU Overlay Control Video data Video Control Other datacontrol | 2010 Wind River. All Rights Reserved.27 Android introduce Stagefright to replace OpenCORE start from clair Either OpenCORE or Stagefright is linked to MediaPlayerService, so application level never know about them Both of them based on same conecption, but take different implementation Stagefright take parser and decode as a whole. Opencore take them as individual node Stagefright take parser, decoder, sink/output as serial operation. Opencore take them as parallel Different Sychronization mechanism Easier for Stagefright to develop parser Adopt OMX from OpenCORE OpenCORE and Stagefright | 2010 Wind River. All Rights Reserved.28 Short point and Stong point OpenCORE more stable and mature, low risk OpenCORE supports more container at and codec Stragefright is more easier to development. OpenCORE and Stagefright | 2010 Wind River. All Rights Reserved.29