[Libav-user] Hardware decoding via libva

[faeem]

Hi All ...

I currently use a media player based on ffplay.c, but have a problem
where HD video playing on an Intel Atom processor is far too slow, and
uses too much CPU time. My understanding is that the problem may be
resolved by using direct hardware decoding via VAAPI/libva. I have
tested VLC compiled with VAAPI support and there is a huge performance
improvement when playing HD video. I'm trying to accomplish something
similar with libav.

Would someone please direct me to documentation and/or some other
explanation of how to perform hardware decoding using VAAPI/libva with
ffmpeg's libav libraries. Code examples would be great. I have no
interest in hardware encoding, only decoding. Unfortunately, several
hours on Google have not yielded any positive results thus far; I'm
probably looking in the wrong places. I have compiled ffmpeg/libav with
VAAPI support. The ffplay.c code has a line that states:

// FIXME use direct rendering

I need to know how to fix that FIXME.

Additional info:
- My media player is based on the 0.11.1 version of ffmpeg's ffplay.c
- My media player has been altered to link against the new 1.0 version
of ffmpeg/libav. However my code is still derived from the ffplay.c
included in ffmpeg 0.11.1.
- The main modification I've made is to render the video via OpenGL
instead of using an overlay. This is a requirement for the project on
which I'm working. The only other major modification is removing the
features of ffplay.c I don't require, which is pretty much everything
aside from the basic decode and render code.
- The code is being developed on Linux. The graphics cards being used
are NVIDIA and Intel i9xxx.

Thanks in advance.

Faeem

[Carl Eugen Hoyos]

Post by faeem
Would someone please direct me to documentation
and/or some other explanation of how to perform
hardware decoding using VAAPI/libva with ffmpeg's
libav libraries. Code examples would be great.

I know of two examples, the va-api code in vlc and
the code in a patch for MPlayer, see for example
http://thread.gmane.org/gmane.comp.video.mplayer.devel/61734/focus=61744

[...]

Post by faeem
// FIXME use direct rendering
I need to know how to fix that FIXME.

This is not related to va-api at all.

Post by faeem
- The main modification I've made is to render the video via OpenGL
instead of using an overlay.

I sounds like a very welcome addition to the FFmpeg
codebase, please consider sending a patch.

Carl Eugen

[Claudio Freire]

Post by Carl Eugen Hoyos

Post by faeem
- The main modification I've made is to render the video via OpenGL
instead of using an overlay.

I sounds like a very welcome addition to the FFmpeg
codebase, please consider sending a patch.

Have in mind that overlays tend to perform better than naively
implemented OpenGL rendering.
It's good to have the option though.

[faeem]

Hi Claudio ...

I'll look into patching ffplay.c with my OpenGL changes and submitting
the code once I've gotten the direct rendering problem sorted out.

I am aware of the performance loss caused by OpenGL. It was chosen over
overlays because it makes adding in special effects much easier,
especially time critical stuff.

Faeem

Post by Claudio Freire

Post by Carl Eugen Hoyos

Post by faeem
- The main modification I've made is to render the video via OpenGL
instead of using an overlay.

I sounds like a very welcome addition to the FFmpeg
codebase, please consider sending a patch.

Have in mind that overlays tend to perform better than naively
implemented OpenGL rendering.
It's good to have the option though.
_______________________________________________
Libav-user mailing list
http://ffmpeg.org/mailman/listinfo/libav-user

[faeem]

I know of two examples, the va-api code in vlc and the code in a patch
for MPlayer, see for example
http://thread.gmane.org/gmane.comp.video.mplayer.devel/61734/focus=61744
[...]

Thanks. I'll be looking into those examples ASAP.

Post by faeem
// FIXME use direct rendering
I need to know how to fix that FIXME.

This is not related to va-api at all.

I selected that //FIXME because it specified "direct rendering", which I
took to mean "handled by the GPU". It seems I was mistaken there.

My conceptual understanding of va-api thus far, within the ffmpeg
framework, is that libavcodec will read an encoded video frame, then use
va-api and the GPU to perform decoding of that frame instead of
performing decoding in software.

The end result will probably be a frame in YUV. I'll need to run the YUV
to RGB conversion on each frame if I'm running OpenGL and this will
still be CPU intensive. I would benefit from the hardware frame decoding
though, and that alone should make a significant difference.

Is this correct?

Faeem

[Carl Eugen Hoyos]

I'll need to run the YUV to RGB conversion on each frame
if I'm running OpenGL and this will still be CPU intensive.

(I don't think this is correct but it is not related to
va-api at all, consider testing mplayer -vo gl, it is
very fast because it does not need CPU conversion of
YUV->RGB.)

[...]

Is this correct?

I am certainly no expert for va-api, but I wonder if
you already realized that ffmpeg and ffplay (the
applications) do not support va-api in any way,
libavcodec contains code that application may use
to decode using va-api, see vlc source code and
the linked MPlayer patch for examples.

Carl Eugen

[faeem]

Thanks for the YUV->RGB tip. I'll look into that.

I think I read on wikipedia that ffmpeg/libavcodec has no va-api
support, but does contain data structures that allow it to be used with
va-api. I figured someone here would have some idea on what to do, and
I've already got some very useful advice and code examples.

So, thanks :)

Faeem

Post by Carl Eugen Hoyos

I'll need to run the YUV to RGB conversion on each frame
if I'm running OpenGL and this will still be CPU intensive.

(I don't think this is correct but it is not related to
va-api at all, consider testing mplayer -vo gl, it is
very fast because it does not need CPU conversion of
YUV->RGB.)
[...]

Is this correct?

I am certainly no expert for va-api, but I wonder if
you already realized that ffmpeg and ffplay (the
applications) do not support va-api in any way,
libavcodec contains code that application may use
to decode using va-api, see vlc source code and
the linked MPlayer patch for examples.
Carl Eugen
_______________________________________________
Libav-user mailing list
http://ffmpeg.org/mailman/listinfo/libav-user

[Oleg]

You're misunderstanding some concepts. There are two operations that can be accelerated by GPU: decoding and yuv->rgb converting. First thing can be achieved by vaapi as you mentioned. Second, by using OpenGL shader(I prefer OpenGL as it's cross-platform. Other option is to use DirectX on Win platform) that will convert YUV->RGB and draw converted frame immediately.

Post by faeem

I know of two examples, the va-api code in vlc and the code in a patch for MPlayer, see for example http://thread.gmane.org/gmane.comp.video.mplayer.devel/61734/focus=61744 [...]

Thanks. I'll be looking into those examples ASAP.

Post by faeem
// FIXME use direct rendering
I need to know how to fix that FIXME.

This is not related to va-api at all.

I selected that //FIXME because it specified "direct rendering", which I took to mean "handled by the GPU". It seems I was mistaken there.
My conceptual understanding of va-api thus far, within the ffmpeg framework, is that libavcodec will read an encoded video frame, then use va-api and the GPU to perform decoding of that frame instead of performing decoding in software.
The end result will probably be a frame in YUV. I'll need to run the YUV to RGB conversion on each frame if I'm running OpenGL and this will still be CPU intensive. I would benefit from the hardware frame decoding though, and that alone should make a significant difference.
Is this correct?
Faeem
_______________________________________________
Libav-user mailing list
http://ffmpeg.org/mailman/listinfo/libav-user

[Xiang, Haihao]

VA API has vaPutSurface() to directly render the decoded frame on a
drawable, you can refer to the code of mplayer vaapi
(git://gitorious.org/vaapi/mplayer.git)

Regards
Haihao

Post by Oleg
You're misunderstanding some concepts. There are two operations that can be accelerated by GPU: decoding and yuv->rgb converting. First thing can be achieved by vaapi as you mentioned. Second, by using OpenGL shader(I prefer OpenGL as it's cross-platform. Other option is to use DirectX on Win platform) that will convert YUV->RGB and draw converted frame immediately.

Post by faeem

I know of two examples, the va-api code in vlc and the code in a patch for MPlayer, see for example http://thread.gmane.org/gmane.comp.video.mplayer.devel/61734/focus=61744 [...]

Thanks. I'll be looking into those examples ASAP.

Post by faeem
// FIXME use direct rendering
I need to know how to fix that FIXME.

This is not related to va-api at all.

I selected that //FIXME because it specified "direct rendering", which I took to mean "handled by the GPU". It seems I was mistaken there.
My conceptual understanding of va-api thus far, within the ffmpeg framework, is that libavcodec will read an encoded video frame, then use va-api and the GPU to perform decoding of that frame instead of performing decoding in software.
The end result will probably be a frame in YUV. I'll need to run the YUV to RGB conversion on each frame if I'm running OpenGL and this will still be CPU intensive. I would benefit from the hardware frame decoding though, and that alone should make a significant difference.
Is this correct?
Faeem
_______________________________________________
Libav-user mailing list
http://ffmpeg.org/mailman/listinfo/libav-user

_______________________________________________
Libav-user mailing list
http://ffmpeg.org/mailman/listinfo/libav-user

[Claudio Freire]

Post by Oleg
You're misunderstanding some concepts. There are two operations that can be accelerated by GPU: decoding and yuv->rgb converting. First thing can be achieved by vaapi as you mentioned. Second, by using OpenGL shader(I prefer OpenGL as it's cross-platform. Other option is to use DirectX on Win platform) that will convert YUV->RGB and draw converted frame immediately.

If you don't use a shader, YUV-RGB conversion not only consumes
cycles. You can actually ignore the cycles it takes on most modern
CPUs. The problem is RGB requires a lot more bandwidth to upload back
into the CPU, and that usually becomes a bottleneck on lower-end
systems for HD content (which requires quite a lot of bandwidth).

Hence my "naively" comment earlier. You don't only need to get the
result in YUV to the shader, but also in downsampled YUV (ie: 4:1:1,
4:2:1, 4:2:2 or whatever format the codec is using internally).