HW video decoding

[markus75]

I just opened this thread for funman to post his info ...

[funman]

Work in progress

Patch for frameworks/base : some debug + look for SEC codecs
Patch for external/opencore : build HW en/decoders (you will need to copy extern_libs_v2/sec/lib/*.so manually in /system/lib)

status: GetpOMX_GetHandle() fails, and I couldn't locate this function (a function pointer is stored in OMXInterface)


I'm going to post some information about video decoding, as on 2.2 builds video decoding is either:

• broken (video do not play)
• choppy/stuttering (video plays too slowly)

On Android, video and audio decoding is based on OpenMAX which permits both software and hardware decoding.

The software decoders are made by PacketVideo, but they are likely much too slow so the CPU is not able to do real time decoding (depending on the codec, the resolution, the audio codec, the bitrates, etc).

The hardware decoders used the DSP (Digital Signal Processor) present on the S3C6410 SoC (System on Chip) to offload some work from the CPU to the DSP, which is much faster to perform some decoding-related tasks.

An hardware implementation of audio/video encoders/decoders for the OMAP3 SoC is present in hardware/ti/omap3 (assuming you cloned criminal's repository), but it is likely useless for us since the S3C6410 SoC on the I5700 doesn't have the TI C64x DSP like the OMAP3.

Software H.264/AVC decoder : in external/opencore/codecs_v2/video/avc_h264/dec/src
It is also present in frameworks/base/media/libstagefright/codecs/avc/dec/src
The code is nearly identical, it was copied in frameworks/base/media/libstagefright on 10 december 2009.

I believe the copy in frameworks/base/media is the one currently being used.

In frameworks/base/media/libstagefright/OMXCodec.cpp there's a list of decoders (kDecoderInfo) to probe when playing a file: the order is important as the first decoder found will be used, so hardware decoders have higher priority. Extract:

    { MEDIA_MIMETYPE_VIDEO_AVC, "OMX.qcom.video.decoder.avc" },
    { MEDIA_MIMETYPE_VIDEO_AVC, "OMX.TI.Video.Decoder" },
    { MEDIA_MIMETYPE_VIDEO_AVC, "AVCDecoder" },

(This list is present in the /system/lib/libstagefright.so binary)

For VIDEO_AVC type (AVC == Advanced Video Codec == H.2.64), we'll try qcom decoder (probably Qualcomm), then TI.Video.Decoder (the Texas Instruments decoder for OMAP3, although it's unlikely to run on our phone), and then the (very slow) software decoder.

Here is what logcat gives me when playing a video file (my build has debug information enabled for libstagefright: I uncommented "#define LOG_NDEBUG 0" in some .cpp files)

V/OMXCodec( 1927): Attempting to allocate OMX node 'OMX.qcom.video.decoder.avc'
V/OMXCodec( 1927): Attempting to allocate OMX node 'OMX.TI.Video.Decoder'
V/OMXCodec( 1927): Successfully allocated software codec 'AVCDecoder'

My first idea was to get this TI.Video.Decoder running but after realizing that the I5700 is not from the OMAP3 family I gave up.


I also tested a 2.1 build from the kitchen, and H264 playback was perfectly smooth (CPU usage for mediaserver binary was around 20%, instead of 65/80% on 2.2).


First thing I saw was that on 2.1 system, there are additional libraries: /system/lib/libsXXXYomxoc.so, with

• XXX = codec (263, 264, aac, ac3, amr, mp4v, vc1, wma)
• Y = e (encoder) or d (decoder)

I tried removing this libs264domxoc.so , and the video player couldn't play my H264 file anymore; so the hardware decoder is likely located in this file.


Now where does it come from? grep told me that it is referenced by libomx_sharedlibrary.so:

$ strings libomx_sharedlibrary.so|grep -E OMX\\. -A 2
OMX.PV.aacenc
audio_encoder.aac
libomx_aacenc_sharedlibrary.so
OMX.PV.avcenc
video_encoder.avc
libomx_avcenc_sharedlibrary.so
OMX.PV.h263enc
video_encoder.h263
libomx_m4venc_sharedlibrary.so
OMX.PV.mpeg4enc
video_encoder.mpeg4
OMX.PV.amrencnb
audio_encoder.amrnb
libomx_amrenc_sharedlibrary.so
OMX.PV.wmadec
libomx_wmadec_sharedlibrary.so
OMX.PV.mp3dec
libomx_mp3dec_sharedlibrary.so
OMX.PV.amrdec
libomx_amrdec_sharedlibrary.so
OMX.PV.aacdec
libomx_aacdec_sharedlibrary.so
OMX.PV.wmvdec
libomx_wmvdec_sharedlibrary.so
OMX.PV.avcdec
libomx_avcdec_sharedlibrary.so
OMX.PV.h263dec
libomx_m4vdec_sharedlibrary.so
OMX.PV.mpeg4dec
OMX.SEC.amrenc
libsamreomxoc.so
OMX.SEC.ac3dec
libsac3domxoc.so
OMX.SEC.wmadec
libswmadomxoc.so
OMX.SEC.aacdec
libsaacdomxoc.so
OMX.SEC.amrdec
libsamrdomxoc.so
OMX.SEC.h263enc
libs263eomxoc.so
OMX.SEC.avcenc
libs264eomxoc.so
OMX.SEC.mpeg4enc
libsmp4veomxoc.so
OMX.SEC.wmvdec
libsvc1domxoc.so
OMX.SEC.h263dec
libs263domxoc.so
OMX.SEC.avcdec
libs264domxoc.so
OMX.SEC.mpeg4dec
libsmp4vdomxoc.so

grep then points to external/opencore/codecs_v2/omx/omx_common/src/pv_omxregistry.cpp and various functions AacRegister(), Mpeg4Register(), AvcRegister() .. which all do:

pCRT->ComponentName = "OMX.PV.mpeg4dec";
...
#if USE_DYNAMIC_LOAD_OMX_COMPONENTS
...
pCRT->SharedLibraryName = "libomx_m4vdec_sharedlibrary.so";
#endif

Supposing the order of the strings in the binary is the same than in source code, "OMX.SEC.avcdec" is the name of the hardware decoder present in libs264domxoc.so


Finally, Leshak just pointed me to a commit which backports Samsung modifications, and these "libs264domxoc.so" and OMX.SEC.* are present, so with this commit video decoding might "just work" already.


In this case I hope this post will at least help someone to understand how video decoding works in a nutshell.

[golecpch]

funman, Did you saw this thread? There is a link to some source of opencore codecs. I wonder if anyone tried to compile it in our froyo.

[funman]

Thanks; I hadn't seen it.

From the top of the logs I see addition of Real codecs

With a quick look it doesn't seem to contain HW decoders, only the software ones.

EDIT: Yes it is HW codec[COLOR="Silver"]

---------- Post added 11-22-2010 at 01:11 AM ---------- Previous post was 11-21-2010 at 10:53 PM ----------

Patches added in 2nd post

[ch33kybutt]

I've compiled latest xvid codec (Xvid.org: Downloads) for armv6 and here's the resultant library "libxvidcore.so" in zip format --> libxvidcore.zip

But it's practically useless as it needs a front-end app to be written to specifically use the library, and afaik none exists for android.

Is there any way to integrate this libxvidcore.so directly into our mediaplayer framework so that it is used by default for xvid format video?

[funman]

I've compiled latest xvid codec (Xvid.org: Downloads) for armv6 and here's the resultant library "libxvidcore.so" in zip format --> libxvidcore.zip

But it's practically useless as it needs a front-end app to be written to specifically use the library, and afaik none exists for android.

Is there any way to integrate this libxvidcore.so directly into our mediaplayer framework so that it is used by default for xvid format video?

xvid doesn't have ARM optimizations so I doubt it will be more useful than the existing m4v decoder by PacketVideo.

The current software decoder is in frameworks/base/media/libstagefright/codecs/m4v_h263/dec , if you really want you could replace the code with xvid code and modify M4vH263Decoder.cpp accordingly

[vflashbirdv]

I've enable avc/mp4 hardware decode now. one problem is can not make mpeg-4 and mp4-avc(h264) file play together. decoder is no problem, but the native PV_Player and StagefrightPlayer each can only support playing one type of them.

[funman]

Do you have a patch or a git repository?

I think 2.1 supports both mpeg-4 ASP and H264 in hardware.

By the way I think PV_Player is software decoder (PV = PacketVideo)

[Krendelrus]

Do you have a patch or a git repository?

afaik, it's https://github.com/vflashbirdv

[vflashbirdv]

I registed github but not used. I'll publish my patch and share my source in sevral days. In AOSP,all media codec are software based, manufactures can add their hardware codec using any method they want, modify PV or StageFright source, or make new component. the leshak patch from samsung is modified the PV,but it's older for 2.1 or even 1.6. I partly ported it to 2.2, can play h263 h264 mp4 now, without DivX and has some bugs atm.