[FFmpeg-devel] [PATCH, v4] lavc/qsvenc: replace assert with error return

[Linjie Fu]

bs->FrameType is not set in MSDK in some cases (mjpeg encode for example),
and assert on a value coming from an external library is not proper.

Add default type check for bs->FrameType, and return invalid data error in function
ff_qsv_encode to avoid using uninitialized value.

Fix #7593.

Signed-off-by: Linjie Fu <***@intel.com>
---
[v2]: Add default bs->FrameType check.
[v3]: Add log message.
[v4]: Fix the indentation.

libavcodec/qsvenc.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c
index 7f4592f878..7877956952 100644
--- a/libavcodec/qsvenc.c
+++ b/libavcodec/qsvenc.c
@@ -1337,8 +1337,13 @@ int ff_qsv_encode(AVCodecContext *avctx, QSVEncContext *q,
pict_type = AV_PICTURE_TYPE_P;
else if (bs->FrameType & MFX_FRAMETYPE_B || bs->FrameType & MFX_FRAMETYPE_xB)
pict_type = AV_PICTURE_TYPE_B;
- else
- av_assert0(!"Uninitialized pict_type!");
+ else if (bs->FrameType == MFX_FRAMETYPE_UNKNOWN) {
+ pict_type = AV_PICTURE_TYPE_NONE;
+ av_log(avctx, AV_LOG_WARNING, "Unkown FrameType, set pict_type to AV_PICTURE_TYPE_NONE.\n");
+ } else {
+ av_log(avctx, AV_LOG_ERROR, "Invalid FrameType:%d.\n", bs->FrameType);
+ return AVERROR_INVALIDDATA;
+ }

#if FF_API_CODED_FRAME
FF_DISABLE_DEPRECATION_WARNINGS

[Li, Zhong]

Of Linjie Fu
Sent: Sunday, December 9, 2018 9:31 PM
Subject: [FFmpeg-devel] [PATCH, v4] lavc/qsvenc: replace assert with error
return
bs->FrameType is not set in MSDK in some cases (mjpeg encode for
bs->example),
and assert on a value coming from an external library is not proper.
Add default type check for bs->FrameType, and return invalid data error in
function ff_qsv_encode to avoid using uninitialized value.
Fix #7593.
---
[v2]: Add default bs->FrameType check.
[v3]: Add log message.
[v4]: Fix the indentation.
libavcodec/qsvenc.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/libavcodec/qsvenc.c b/libavcodec/qsvenc.c index
7f4592f878..7877956952 100644
--- a/libavcodec/qsvenc.c
+++ b/libavcodec/qsvenc.c
@@ -1337,8 +1337,13 @@ int ff_qsv_encode(AVCodecContext *avctx, QSVEncContext *q,
pict_type = AV_PICTURE_TYPE_P;
else if (bs->FrameType & MFX_FRAMETYPE_B || bs->FrameType & MFX_FRAMETYPE_xB)
pict_type = AV_PICTURE_TYPE_B;
- else
- av_assert0(!"Uninitialized pict_type!");
+ else if (bs->FrameType == MFX_FRAMETYPE_UNKNOWN) {
+ pict_type = AV_PICTURE_TYPE_NONE;
+ av_log(avctx, AV_LOG_WARNING, "Unkown FrameType, set
pict_type to AV_PICTURE_TYPE_NONE.\n");
+ } else {
+ av_log(avctx, AV_LOG_ERROR, "Invalid FrameType:%d.\n", bs->FrameType);
+ return AVERROR_INVALIDDATA;
+ }
#if FF_API_CODED_FRAME
FF_DISABLE_DEPRECATION_WARNINGS
--
2.17.1

Verified and applied with an updated commit message to highlight this is a regression.