rtpdec_h264.c RTP to annex-b patch

[Matthew Szatmary]

Excuse me if i'm missing something, This is my first time submitting a patch to this group.
This patch formats h264 RTP streams as annex-b format. This allows for transmuxing without transcoding (e.g "-vcodec copy") RTP and RTSP streams.

[Michael Niedermayer]

Post by Matthew Szatmary
Excuse me if i'm missing something, This is my first time submitting a patch to this group.
This patch formats h264 RTP streams as annex-b format. This allows for transmuxing without transcoding (e.g "-vcodec copy") RTP and RTSP streams.

Do we have a maintainer for this file, or volunteer to become
maintainer ?

about the patch, if the output isnt compliant to either of the
formats then yes it should be fixed.
note, tabs are forbiden in ffmpeg git


[...]

[Martin Storsjö]

Post by Michael Niedermayer

Post by Matthew Szatmary
Excuse me if i'm missing something, This is my first time submitting a patch to this group.
This patch formats h264 RTP streams as annex-b format. This allows for transmuxing without transcoding (e.g "-vcodec copy") RTP and RTSP streams.

Do we have a maintainer for this file, or volunteer to become
maintainer ?
about the patch, if the output isnt compliant to either of the
formats then yes it should be fixed.
note, tabs are forbiden in ffmpeg git

FWIW, to me, it seems that the code already produces data in annex b
format (with 3 byte startcodes, which should be legal, right?). This patch
changes things so that SPS/PPS aren't exported via extradata as earlier,
but are prepended to packets with nal type 5.

If this is required for stream copy to some certain format, it's probably
better to do it either via a bitstream filter, or to make the target muxer
accept this format as input. Since if this currently fixes one format, it
will probably just as well break some other format which expects this
format.

Also, I much prefer having the SPS/PPS out of band if they actually are
transmitted in that format (as they are, in RTP) than mixed into the
stream.

// Martin

[Matthew Szatmary]

The 3 byre start codes and out of band SPS/PPS format is legal, but the only thing in the ffmpeg stack that can parse that is the h264 decoder itself. I could have made this a bit stream filter, but it would do the exact same thing and require extra code / CLI options from the user. Maybe this solution is not ideal for everyone, but currently it is not possible to write an H.264 RTSP stream without transcoding it. This fixes that.


I'll fix the tabs, and can volunteer to be the maintainer of this file if necessary.

Post by Michael Niedermayer

Post by Matthew Szatmary
Excuse me if i'm missing something, This is my first time submitting a patch to this group.
This patch formats h264 RTP streams as annex-b format. This allows for transmuxing without transcoding (e.g "-vcodec copy") RTP and RTSP streams.

Do we have a maintainer for this file, or volunteer to become
maintainer ?
about the patch, if the output isnt compliant to either of the
formats then yes it should be fixed.
note, tabs are forbiden in ffmpeg git

FWIW, to me, it seems that the code already produces data in annex b format (with 3 byte startcodes, which should be legal, right?). This patch changes things so that SPS/PPS aren't exported via extradata as earlier, but are prepended to packets with nal type 5.
If this is required for stream copy to some certain format, it's probably better to do it either via a bitstream filter, or to make the target muxer accept this format as input. Since if this currently fixes one format, it will probably just as well break some other format which expects this format.
Also, I much prefer having the SPS/PPS out of band if they actually are transmitted in that format (as they are, in RTP) than mixed into the stream.
// Martin
_______________________________________________
ffmpeg-devel mailing list
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

[Martin Storsjö]

Post by Matthew Szatmary
The 3 byre start codes and out of band SPS/PPS format is legal, but the
only thing in the ffmpeg stack that can parse that is the h264 decoder
itself. I could have made this a bit stream filter, but it would do the
exact same thing and require extra code / CLI options from the user.
Maybe this solution is not ideal for everyone, but currently it is not
possible to write an H.264 RTSP stream without transcoding it. This
fixes that.

You didn't say which format you tried to write it into - I'm pretty sure
you can write the output from this into mov/mp4 without problems, which
wouldn't work with this applied.

// Martin

[Matthew Szatmary]

using -vcodec copy I tried outputting as mp4, mkv, fly and mpegts (with and without -bsf h264_mp4toannexb). None of them worked before this patch. They all work after this patch is applied

Post by Matthew Szatmary
The 3 byre start codes and out of band SPS/PPS format is legal, but the only thing in the ffmpeg stack that can parse that is the h264 decoder itself. I could have made this a bit stream filter, but it would do the exact same thing and require extra code / CLI options from the user. Maybe this solution is not ideal for everyone, but currently it is not possible to write an H.264 RTSP stream without transcoding it. This fixes that.

You didn't say which format you tried to write it into - I'm pretty sure you can write the output from this into mov/mp4 without problems, which wouldn't work with this applied.
// Martin
_______________________________________________
ffmpeg-devel mailing list
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

[Martin Storsjö]

Post by Matthew Szatmary
using -vcodec copy I tried outputting as mp4, mkv, fly and mpegts (with
and without -bsf h264_mp4toannexb). None of them worked before this
patch. They all work after this patch is applied

Which errors did you get when they didn't work?

The h264_mp4toannexb bsf isn't exactly what's needed in this case - the
data is in annex b format already, but with SPS/PPS as extradata instead
of inline in the stream. (This is exactly the same format as libx264
outputs if encoding for mp4 files, when the global header flag is set,
FWIW.) Not sure if there currently is any bitstream filter to do exactly
this conversion (removing extradata and embedding it into the stream)
though.

In my tests, remuxing from rtp/h264 into mp4 works just fine with the
current code, try e.g. this:

ffmpeg -allowed_media_types video -rtsp_transport tcp -i
rtsp://albin.abo.fi:8554/sample_h264_100kbit.mp4 -vcodec copy -an -y
test.mp4

The video produced with this command currently works better than what I
get with your patch applied.

However in other cases, I do see the issue you're facing with mkv, but
I'm not really sure it's a more correct solution even if some cases might
work better.

Currently, the rtp depacketizer sets timestamps for all packets, but it
seems that the h264 parser clears this from the first frame, so that it
has dts=pts=N/A. With the current code, this frame has keyframe=1 set, and
the mkv muxer will fail to write it with "Can't write packet with unknown
timestamp". With your patch applied, this frame is marked as keyframe=0 by
the parser, and ffmpeg.c won't start to try to write the packets to mkv
until you run into the next keyframe, which has a timestamp. (If you try
your patch with -copyinkf you'll get the same error again.)

So it seems to me that with SPS/PPS inline, the h264 parser behaves
slightly differently in what it outputs (especially when it doesn't set
the keyframe flag for the first packets), which happens to work better
when muxing.

As for muxing into mpegts, you will probably need either your patch or a
bsf to move the extradata into the stream though.

// Martin

[Michael Niedermayer]

Post by Martin Storsjö

Post by Matthew Szatmary
using -vcodec copy I tried outputting as mp4, mkv, fly and mpegts
(with and without -bsf h264_mp4toannexb). None of them worked
before this patch. They all work after this patch is applied

Which errors did you get when they didn't work?
The h264_mp4toannexb bsf isn't exactly what's needed in this case -
the data is in annex b format already, but with SPS/PPS as extradata
instead of inline in the stream. (This is exactly the same format as
libx264 outputs if encoding for mp4 files, when the global header
flag is set, FWIW.) Not sure if there currently is any bitstream
filter to do exactly this conversion (removing extradata and
embedding it into the stream) though.

dump_extra maybe

but in how far that would produce valid annex b iam not sure.
Our decoder will handle it of course but that doesnt mean its compliant
to the spec.


[...]

Post by Martin Storsjö
Currently, the rtp depacketizer sets timestamps for all packets, but
it seems that the h264 parser clears this from the first frame, so
that it has dts=pts=N/A.

that sounds like a bug

Post by Martin Storsjö
With the current code, this frame has
keyframe=1 set, and the mkv muxer will fail to write it with "Can't
write packet with unknown timestamp". With your patch applied, this
frame is marked as keyframe=0 by the parser, and ffmpeg.c won't

Is it or is it not a keyframe ?
either way this too sounds like theres a bug somewhere


[...]

[Martin Storsjö]

Post by Michael Niedermayer

Post by Martin Storsjö
The h264_mp4toannexb bsf isn't exactly what's needed in this case -
the data is in annex b format already, but with SPS/PPS as extradata
instead of inline in the stream. (This is exactly the same format as
libx264 outputs if encoding for mp4 files, when the global header
flag is set, FWIW.) Not sure if there currently is any bitstream
filter to do exactly this conversion (removing extradata and
embedding it into the stream) though.

dump_extra maybe
but in how far that would produce valid annex b iam not sure.
Our decoder will handle it of course but that doesnt mean its compliant
to the spec.

Hmm, that might actually work.

Post by Michael Niedermayer
[...]

Post by Martin Storsjö
Currently, the rtp depacketizer sets timestamps for all packets, but
it seems that the h264 parser clears this from the first frame, so
that it has dts=pts=N/A.

that sounds like a bug

Post by Martin Storsjö
With the current code, this frame has
keyframe=1 set, and the mkv muxer will fail to write it with "Can't
write packet with unknown timestamp". With your patch applied, this
frame is marked as keyframe=0 by the parser, and ffmpeg.c won't

Is it or is it not a keyframe ?
either way this too sounds like theres a bug somewhere

I do think it's a keyframe.

The stream is available publicly in the url as in the previous mail
(rtsp://albin.abo.fi:8554/sample_h264_100kbit.mp4), if you want to have a
look.

// Martin

[Michael Niedermayer]

Post by Martin Storsjö

Post by Michael Niedermayer

Post by Martin Storsjö
The h264_mp4toannexb bsf isn't exactly what's needed in this case -
the data is in annex b format already, but with SPS/PPS as extradata
instead of inline in the stream. (This is exactly the same format as
libx264 outputs if encoding for mp4 files, when the global header
flag is set, FWIW.) Not sure if there currently is any bitstream
filter to do exactly this conversion (removing extradata and
embedding it into the stream) though.

dump_extra maybe
but in how far that would produce valid annex b iam not sure.
Our decoder will handle it of course but that doesnt mean its compliant
to the spec.

Hmm, that might actually work.

Post by Michael Niedermayer
[...]

Post by Martin Storsjö
Currently, the rtp depacketizer sets timestamps for all packets, but
it seems that the h264 parser clears this from the first frame, so
that it has dts=pts=N/A.

that sounds like a bug

Post by Martin Storsjö
With the current code, this frame has
keyframe=1 set, and the mkv muxer will fail to write it with "Can't
write packet with unknown timestamp". With your patch applied, this
frame is marked as keyframe=0 by the parser, and ffmpeg.c won't

Is it or is it not a keyframe ?
either way this too sounds like theres a bug somewhere

I do think it's a keyframe.
The stream is available publicly in the url as in the previous mail
(rtsp://albin.abo.fi:8554/sample_h264_100kbit.mp4), if you want to
have a look.

hmm
this doesnt look good with the patch, i get 2 startcodes
0 0 0 1 0 0 0 1
then duplicate SPS & PPS


[...]

[Matthew Szatmary]

The stream I'm testing with is generated by a security camera. (And ffmpeg completely chokes). This patch works perfectly for my stream. Thank you for testing an providing a sample stream. I'll modify the patch so it works with this source as well.

Please ignore this patch for now. I'll post a new version soon.

Sent from my phone

Post by Michael Niedermayer

Post by Martin Storsjö

Post by Michael Niedermayer

Post by Martin Storsjö
The h264_mp4toannexb bsf isn't exactly what's needed in this case -
the data is in annex b format already, but with SPS/PPS as extradata
instead of inline in the stream. (This is exactly the same format as
libx264 outputs if encoding for mp4 files, when the global header
flag is set, FWIW.) Not sure if there currently is any bitstream
filter to do exactly this conversion (removing extradata and
embedding it into the stream) though.

dump_extra maybe
but in how far that would produce valid annex b iam not sure.
Our decoder will handle it of course but that doesnt mean its compliant
to the spec.

Hmm, that might actually work.

Post by Michael Niedermayer
[...]

Post by Martin Storsjö
Currently, the rtp depacketizer sets timestamps for all packets, but
it seems that the h264 parser clears this from the first frame, so
that it has dts=pts=N/A.

that sounds like a bug

Post by Martin Storsjö
With the current code, this frame has
keyframe=1 set, and the mkv muxer will fail to write it with "Can't
write packet with unknown timestamp". With your patch applied, this
frame is marked as keyframe=0 by the parser, and ffmpeg.c won't

Is it or is it not a keyframe ?
either way this too sounds like theres a bug somewhere

I do think it's a keyframe.
The stream is available publicly in the url as in the previous mail
(rtsp://albin.abo.fi:8554/sample_h264_100kbit.mp4), if you want to
have a look.

hmm
this doesnt look good with the patch, i get 2 startcodes
0 0 0 1 0 0 0 1
then duplicate SPS & PPS
[...]
--
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
The worst form of inequality is to try to make unequal things equal.
-- Aristotle
_______________________________________________
ffmpeg-devel mailing list
http://ffmpeg.org/mailman/listinfo/ffmpeg-devel

[Michael Niedermayer]

Post by Matthew Szatmary
The 3 byre start codes and out of band SPS/PPS format is legal, but the only thing in the ffmpeg stack that can parse that is the h264 decoder itself. I could have made this a bit stream filter, but it would do the exact same thing and require extra code / CLI options from the user. Maybe this solution is not ideal for everyone, but currently it is not possible to write an H.264 RTSP stream without transcoding it. This fixes that.
I'll fix the tabs, and can volunteer to be the maintainer of this file if necessary.

Yes, i would be very happy if you would maintain this file!
Simplest would likely be if you clone the ffmpeg repo somewhere (like
github) and when you have some commits which with you are happy, tell
me to pull them in.

Post by Matthew Szatmary

Post by Michael Niedermayer

Post by Matthew Szatmary
Excuse me if i'm missing something, This is my first time submitting a patch to this group.
This patch formats h264 RTP streams as annex-b format. This allows for transmuxing without transcoding (e.g "-vcodec copy") RTP and RTSP streams.

Do we have a maintainer for this file, or volunteer to become
maintainer ?
about the patch, if the output isnt compliant to either of the
formats then yes it should be fixed.
note, tabs are forbiden in ffmpeg git

FWIW, to me, it seems that the code already produces data in annex b format (with 3 byte startcodes, which should be legal, right?). This patch changes things so that SPS/PPS aren't exported via extradata as earlier, but are prepended to packets with nal type 5.

annex B:
zero_byte is a single byte equal to 0x00.
When any of the following conditions are fulfilled, the zero_byte syntax element shall be present:
­ the nal_unit_type within the nal_unit( ) is equal to 7 (sequence parameter set) or 8 (picture parameter set),
­ the byte stream NAL unit syntax structure contains the first NAL unit of an access unit in decoding order, as
specified by subclause 7.4.1.2.3.

[...]