B frames and avi files

[D Richard Felker III]

I'm working on mplayer g2, and trying to work out the horrible
nightmare of pts with B-frames in avi files. From what I can gather,
B-frames in avi files seem to be stored in codec order, meaning that
their frame numbers within the avi file do _not_ represent their
display order. This is very troubling, since it makes it nearly
impossible to get a correct timestamp after seeking!

So far, I've only come across one way to accurately determine frame
numbers in display order: the display-order frame number of a
reference frame (I/P) is the coded-order frame number of the next
reference frame, minus 1. However, I'm concerned that there's no way
to see where the next reference frame is before lavc decodes the
current one.

There's a related problem with B-frames in MPlayer, the notorious
glitch-after-seeking bug. In case you're not familiar with it, MPlayer
will show a bogus blocky mix of the scene before the seek and the new
scene after the seek for a few frames after seeking, when playing
files with B-frames. I'll explain the cause:

Suppose after seeking, the next 4 frames (coded order) are IBBP. The
two B-frames actually belong before the I-frame, and should not be
shown after seeking. But the decoder (lavc or libmpeg2) has no way of
knowing this, and thinks the two B-frames belong between the pre-seek
reference frame and the new I-frame. Thus, broken output.

The only real fix I can think of for this problem, is to provide a
method for the caller to notify lavc that a seek has taken place, and
to "flush the buffers". After such a notification, all B-frames should
be dropped until the second reference frame is found.

If in addition to this, the first reference frame after the seek (the
I-frame) is not decoded until the second reference frame is found,
then it becomes possible to obtain an accurate frame number in avi
files! The (display-order) frame number of the I-frame you seek to is
exactly the (coded-order) frame number of the frame that causes it to
be decoded, minus 1.

Could someone who understands these issues comment? Maybe other
methods of dealing with these problems are already known and I'm just
ignorant. Feel free to flame if that's the case. :)


Rich



P.S. Maybe what I just said can be emulated with hurry_up flag and
wrappers...?


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Fabrice Bellard]

Post by D Richard Felker III
I'm working on mplayer g2, and trying to work out the horrible
nightmare of pts with B-frames in avi files. From what I can gather,
B-frames in avi files seem to be stored in codec order, meaning that
their frame numbers within the avi file do _not_ represent their
display order. This is very troubling, since it makes it nearly
impossible to get a correct timestamp after seeking!

Do you have an example of such AVIs ? I am also interested to look at
this as I am adding support for B frames in libavformat.

Post by D Richard Felker III
So far, I've only come across one way to accurately determine frame
numbers in display order: the display-order frame number of a
reference frame (I/P) is the coded-order frame number of the next
reference frame, minus 1. However, I'm concerned that there's no way
to see where the next reference frame is before lavc decodes the
current one.
There's a related problem with B-frames in MPlayer, the notorious
glitch-after-seeking bug. In case you're not familiar with it, MPlayer
will show a bogus blocky mix of the scene before the seek and the new
scene after the seek for a few frames after seeking, when playing
Suppose after seeking, the next 4 frames (coded order) are IBBP. The
two B-frames actually belong before the I-frame, and should not be
shown after seeking. But the decoder (lavc or libmpeg2) has no way of
knowing this, and thinks the two B-frames belong between the pre-seek
reference frame and the new I-frame. Thus, broken output.
The only real fix I can think of for this problem, is to provide a
method for the caller to notify lavc that a seek has taken place, and
to "flush the buffers". After such a notification, all B-frames should
be dropped until the second reference frame is found.
If in addition to this, the first reference frame after the seek (the
I-frame) is not decoded until the second reference frame is found,
then it becomes possible to obtain an accurate frame number in avi
files! The (display-order) frame number of the I-frame you seek to is
exactly the (coded-order) frame number of the frame that causes it to
be decoded, minus 1.
Could someone who understands these issues comment? Maybe other
methods of dealing with these problems are already known and I'm just
ignorant. Feel free to flame if that's the case. :)

The safe solution is as you say to avoid decoding B frames until you
have two reference frames, so the codec flush must ensure that.

But it can be more complicated, at least in MPEG: you can find cases
where even if you have the pattern IBBP after a GOP, you can decode the
B frames because they have forward motion vectors only. The 'closed_gop'
bit in the GOP header tells you that.

Fabrice.



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[D Richard Felker III]

Post by Fabrice Bellard

Post by D Richard Felker III
I'm working on mplayer g2, and trying to work out the horrible
nightmare of pts with B-frames in avi files. From what I can gather,
B-frames in avi files seem to be stored in codec order, meaning that
their frame numbers within the avi file do _not_ represent their
display order. This is very troubling, since it makes it nearly
impossible to get a correct timestamp after seeking!

Do you have an example of such AVIs ? I am also interested to look at
this as I am adding support for B frames in libavformat.

Just use mencoder (or maybe ffmpeg?) with libavcodec set to output B
frames, and it will make one for you... Plenty people make files this
way and they seem to work on windows players.

Post by Fabrice Bellard

Post by D Richard Felker III
Could someone who understands these issues comment? Maybe other
methods of dealing with these problems are already known and I'm just
ignorant. Feel free to flame if that's the case. :)

The safe solution is as you say to avoid decoding B frames until you
have two reference frames, so the codec flush must ensure that.
But it can be more complicated, at least in MPEG: you can find cases
where even if you have the pattern IBBP after a GOP, you can decode the
B frames because they have forward motion vectors only. The 'closed_gop'
bit in the GOP header tells you that.

Hmm, then our 'safe' solution isn't really safe. There has to be some
way the codec can tell from the headers whether the B frame belongs
before or after the I frame, right...??

Rich



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Mean]

Post by D Richard Felker III
If in addition to this, the first reference frame after the seek (the
I-frame) is not decoded until the second reference frame is found,
then it becomes possible to obtain an accurate frame number in avi
files! The (display-order) frame number of the I-frame you seek to is
exactly the (coded-order) frame number of the frame that causes it to
be decoded, minus 1.

There is a bit more to that.

Depending if B frames are present or not (low delay flag in header), the
codec (lavcodec or
mpeg2dec) will hold the frames for possibly 2 ticks before sending them out
to guarantee one frame in, one frame out and avoid possible variable
latency.


There is another case that breaks the rule, divx (or xvid) PB frames that
packs the B frame with its refernce frame and insert a null frame afterward

xxx I B P B PB xxxx
=> xx I (BP) 0 (BP) 0 xxxx

That forces the decision to be made at codec level and cause a pseudo
low delay.
That is B fame but with low delay.

you need to decode the null frame as it contains the second part of the
PB frame.

So to sum it up :
- low delay or PB frames : 0 delay
- real B frames : 2 frames delay

That makes frame accuracy / timestamp difficult








-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[D Richard Felker III]

Post by Mean

Post by D Richard Felker III
If in addition to this, the first reference frame after the seek (the
I-frame) is not decoded until the second reference frame is found,
then it becomes possible to obtain an accurate frame number in avi
files! The (display-order) frame number of the I-frame you seek to is
exactly the (coded-order) frame number of the frame that causes it to
be decoded, minus 1.

There is a bit more to that.
Depending if B frames are present or not (low delay flag in header), the
codec (lavcodec or
mpeg2dec) will hold the frames for possibly 2 ticks before sending them out
to guarantee one frame in, one frame out and avoid possible variable
latency.
There is another case that breaks the rule, divx (or xvid) PB frames that
packs the B frame with its refernce frame and insert a null frame afterward
xxx I B P B PB xxxx
=> xx I (BP) 0 (BP) 0 xxxx
That forces the decision to be made at codec level and cause a pseudo
low delay.
That is B fame but with low delay.
you need to decode the null frame as it contains the second part of the
PB frame.
- low delay or PB frames : 0 delay
- real B frames : 2 frames delay
That makes frame accuracy / timestamp difficult

How disgusting.... A few questions:

Is there any way to distinguish the two types? Could lavc somehow
provide this info to the caller?

Does the divx/xvid method allow more than one B frame?
i.e. I (BBP) 0 0 (BPP) 0 0 ...?

As stupid as it is, I'm inclined to think that the divx/xvid method is
the only possible correct way of storing video with B frames in an avi
file, and mencoder is stupid for storing the frames the way it does...
But, in any case, I want MPlayer G2 to be able to handle _all_ files
with 100% exact pts, so some workaround needs to be found...

Rich



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[D Richard Felker III]

Post by D Richard Felker III

Post by Mean
There is another case that breaks the rule, divx (or xvid) PB frames that
packs the B frame with its refernce frame and insert a null frame afterward
xxx I B P B PB xxxx
=> xx I (BP) 0 (BP) 0 xxxx
That forces the decision to be made at codec level and cause a pseudo
low delay.
That is B fame but with low delay.
you need to decode the null frame as it contains the second part of the
PB frame.
- low delay or PB frames : 0 delay
- real B frames : 2 frames delay
That makes frame accuracy / timestamp difficult

Is there any way to distinguish the two types? Could lavc somehow
provide this info to the caller?
Does the divx/xvid method allow more than one B frame?
i.e. I (BBP) 0 0 (BPP) 0 0 ...?
As stupid as it is, I'm inclined to think that the divx/xvid method is
the only possible correct way of storing video with B frames in an avi

Actually, now that I think about it, I think it's probably broken too.
How do you store the B frames immediately before an I frame? Something
like this?

... (BP) 0 (BP) 0 (BI) 0 (BP) 0 ...
^^^^

If that's the case, then what is the player expected to do when
seeking to this keyframe? If it decodes the B frame it will get junk,
so it has to skip it. But then it also has to ignore the following
null-frame, which would usually be treated as repeat-prev-frame. I
guess this works, but it's incredibly ugly. Or do xvid/divx just not
output B frames before an I frame?

Rich



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Mean]

Post by D Richard Felker III
Actually, now that I think about it, I think it's probably broken too.
How do you store the B frames immediately before an I frame? Something
like this?
... (BP) 0 (BP) 0 (BI) 0 (BP) 0 ...
^^^^
If that's the case, then what is the player expected to do when
seeking to this keyframe? If it decodes the B frame it will get junk,
so it has to skip it. But then it also has to ignore the following
null-frame, which would usually be treated as repeat-prev-frame. I
guess this works, but it's incredibly ugly. Or do xvid/divx just not
output B frames before an I frame?

In that case it will probably pop the I frame 2 times
You now see what i meant by double-guessing what the codec will do.

I much prefer the mencoder/mpeg way as it is the same stuff as for mpeg1/2

To solve this, i do it like that, but i doubt it would be acceptable for
mplayer
- Detect if the stream has B frame by decoding a few frames
- If yes, reorder the frames to DTS order
- For playback always use LOW_DELAY flag.

If the stream has no B frame, do nothing special
If it has B frame, feed the next reference and store it, then B frames
When we reach the reference frame, we just pop it back

It has a couple of drawbacks ( need to know frame type, need to have a
index etc...) that is not compatible with mplayer
capabilities (streaming etc...) but it gives perfect frame accuracy and
is the same for avi and mpeg.






-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[D Richard Felker III]

Post by Mean

Post by D Richard Felker III
Actually, now that I think about it, I think it's probably broken too.
How do you store the B frames immediately before an I frame? Something
like this?
... (BP) 0 (BP) 0 (BI) 0 (BP) 0 ...
^^^^
If that's the case, then what is the player expected to do when
seeking to this keyframe? If it decodes the B frame it will get junk,
so it has to skip it. But then it also has to ignore the following
null-frame, which would usually be treated as repeat-prev-frame. I
guess this works, but it's incredibly ugly. Or do xvid/divx just not
output B frames before an I frame?

In that case it will probably pop the I frame 2 times
You now see what i meant by double-guessing what the codec will do.

Yes...

Post by Mean
I much prefer the mencoder/mpeg way as it is the same stuff as for mpeg1/2
To solve this, i do it like that, but i doubt it would be acceptable for
mplayer
- Detect if the stream has B frame by decoding a few frames
- If yes, reorder the frames to DTS order

They're in DTS order to begin with. That's actually the problem. :)
I guess that means we don't need to detect if the stream has B frames.

Post by Mean
- For playback always use LOW_DELAY flag.

What exactly does the LOW_DELAY flag do? From what I said to begin
with, it sounds like it's an advantage not to have low_delay.

Post by Mean
If the stream has no B frame, do nothing special
If it has B frame, feed the next reference and store it, then B frames
When we reach the reference frame, we just pop it back
It has a couple of drawbacks ( need to know frame type, need to have a
index etc...) that is not compatible with mplayer

How do you know frame type in advance? It's not in the index.
BTW, if you start decoding from the beginning and never seek,
everything is easy. You just assign frame numbers as lavc outputs the
frames. The _only_ difficulty is figuring out the correct starting
frame number after a seek.

Post by Mean
capabilities (streaming etc...) but it gives perfect frame accuracy and
is the same for avi and mpeg.

Handling mpeg correctly is already trivial, so we'll stick with avi if
that's ok...

Here, I have one possible working method for recovering pts after
seeking:

Decode first I frame --> output with unknown frameno
Skip subsequent B frames
Decode the next I/P frame --> output with unknown frameno
Decode the next frame --> output with frameno = coded_frameno of
previous frame

Unfortunately you have to output 2 frames without knowing their pts,
which is rather bad. Hopefully there's a better way. BTW I think this
works with both divx-packed and mencoder-style B frames.


Rich



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Mean]

Post by D Richard Felker III

Post by Mean
I much prefer the mencoder/mpeg way as it is the same stuff as for mpeg1/2
To solve this, i do it like that, but i doubt it would be acceptable for
mplayer
- Detect if the stream has B frame by decoding a few frames
- If yes, reorder the frames to DTS order

They're in DTS order to begin with. That's actually the problem. :)
I guess that means we don't need to detect if the stream has B frames.

oops, reorder them to presentation order, not decoding order.
That way it also solves the problem of seeking as if you seek to I0

Decoding order
I0 B0 B1 P xxx

Presentation order
B0 B1 I0 B B P
^Seek here
You don't have to bother about b0 and b1 at all

Post by D Richard Felker III

Post by Mean
- For playback always use LOW_DELAY flag.

What exactly does the LOW_DELAY flag do? From what I said to begin
with, it sounds like it's an advantage not to have low_delay.

Low delay tell the code to immediatly ouput the decoded frame without
handling the PTS/DTS logic.

If you feed
I P B B , you will get I P B B immediatly

It depend if you want to let the codec do all by itself or not.
If yes, do not touch low_delay, you will run into trouble

If no, force it and handle the PTS/DTS logic at application level as i do.

Post by D Richard Felker III

Post by Mean
If the stream has no B frame, do nothing special
If it has B frame, feed the next reference and store it, then B frames
When we reach the reference frame, we just pop it back
It has a couple of drawbacks ( need to know frame type, need to have a
index etc...) that is not compatible with mplayer

How do you know frame type in advance? It's not in the index.
BTW, if you start decoding from the beginning and never seek,
everything is easy. You just assign frame numbers as lavc outputs the
frames. The _only_ difficulty is figuring out the correct starting
frame number after a seek.

Simple, i use an extra flag in the index to say if it is a B frame.
If the flag is not here and the decoder tells me it is a B frame
, i rederive the type by using HURRY_UP=5
+ (rejected) patch to set correct frame type for B frame.
It takes 6 seconds to rederive a 2 hours movie.

But again, it is probably not acceptable for mplayer.

Post by D Richard Felker III

Post by Mean
capabilities (streaming etc...) but it gives perfect frame accuracy and
is the same for avi and mpeg.

Handling mpeg correctly is already trivial, so we'll stick with avi if
that's ok...

Ok, but it is exactly the same thing. It would also solve the broken
frames after seeking for mpeg.
It is exactly the same problem.






-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[D Richard Felker III]

Post by Mean

Post by D Richard Felker III

Post by Mean
- For playback always use LOW_DELAY flag.

What exactly does the LOW_DELAY flag do? From what I said to begin
with, it sounds like it's an advantage not to have low_delay.

Low delay tell the code to immediatly ouput the decoded frame without
handling the PTS/DTS logic.
If you feed
I P B B , you will get I P B B immediatly
It depend if you want to let the codec do all by itself or not.
If yes, do not touch low_delay, you will run into trouble
If no, force it and handle the PTS/DTS logic at application level as i do.

Thanks for the explanation.

Post by Mean

Post by D Richard Felker III
How do you know frame type in advance? It's not in the index.
BTW, if you start decoding from the beginning and never seek,
everything is easy. You just assign frame numbers as lavc outputs the
frames. The _only_ difficulty is figuring out the correct starting
frame number after a seek.

Simple, i use an extra flag in the index to say if it is a B frame.
If the flag is not here and the decoder tells me it is a B frame
, i rederive the type by using HURRY_UP=5
+ (rejected) patch to set correct frame type for B frame.
It takes 6 seconds to rederive a 2 hours movie.
But again, it is probably not acceptable for mplayer.

Yes, definitely not... 100ms is not acceptable delay for mplayer. BTW,
you're talking about an ultra-fast hard drive. Reindexing a movie on
my hd takes at least 30-40 seconds, and it takes a good 3-4 minutes
from cdrom.

BTW, why was your hurry_up patch rejected? It sounds useful..

Post by Mean

Post by D Richard Felker III
Handling mpeg correctly is already trivial, so we'll stick with avi if
that's ok...

Ok, but it is exactly the same thing. It would also solve the broken
frames after seeking for mpeg.
It is exactly the same problem.

IMO not at all. The problem with AVI is that you don't have the
information to reassign the frame numbers until you've already decoded
them, and then it's too late. With mpeg container, you already have
proper DTS and PTS, and any good codec will give you frames in the
right (display) order and with the correct timestamps.

As for the broken frames after seeking, it's just a matter of failing
to call the codec's flush() function.

BTW....what happens when you put xvid w/B-frames in .ogm? I hate to
think about it....

Rich



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Mean]

Post by D Richard Felker III

Post by Mean

Post by D Richard Felker III
How do you know frame type in advance? It's not in the index.
BTW, if you start decoding from the beginning and never seek,
everything is easy. You just assign frame numbers as lavc outputs the
frames. The _only_ difficulty is figuring out the correct starting
frame number after a seek.

Simple, i use an extra flag in the index to say if it is a B frame.
If the flag is not here and the decoder tells me it is a B frame
, i rederive the type by using HURRY_UP=5
+ (rejected) patch to set correct frame type for B frame.
It takes 6 seconds to rederive a 2 hours movie.
But again, it is probably not acceptable for mplayer.

Yes, definitely not... 100ms is not acceptable delay for mplayer. BTW,
you're talking about an ultra-fast hard drive. Reindexing a movie on
my hd takes at least 30-40 seconds, and it takes a good 3-4 minutes
from cdrom.

Yes, you are right. To be accurate, the decoding time is neglectable as
with hurry_up=5 it only decodes the VOP headers, so it is almost
instantaneous. you have to add the access time to the media ,
which is not that bad as you already have a valid index.

Post by D Richard Felker III
BTW, why was your hurry_up patch rejected? It sounds useful..

It is useful only when low_delay flag was set/forced.
It is not common to force low_delay when b frame are present.

Post by D Richard Felker III

Post by Mean

Post by D Richard Felker III
Handling mpeg correctly is already trivial, so we'll stick with avi if
that's ok...

Ok, but it is exactly the same thing. It would also solve the broken
frames after seeking for mpeg.
It is exactly the same problem.

IMO not at all. The problem with AVI is that you don't have the
information to reassign the frame numbers until you've already decoded
them, and then it's too late. With mpeg container, you already have
proper DTS and PTS, and any good codec will give you frames in the
right (display) order and with the correct timestamps.

Yes. You can deal with that problem 2 ways :
- handling presentation order at application level
- using pts/dts

The 1st one is easier for avi, the second one for mpeg.
Using pts/dts is better as it ease sync with audio.

Post by D Richard Felker III
As for the broken frames after seeking, it's just a matter of failing
to call the codec's flush() function.

yes, and handling the delay if you don"t have pts/dts.

Post by D Richard Felker III
BTW....what happens when you put xvid w/B-frames in .ogm? I hate to
think about it....

Same story, you reorder but first create an index.
Ogm is fine to play sequentially, horrible to edit.




-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[D Richard Felker III]

Post by Mean

Post by D Richard Felker III

Post by Mean

Post by D Richard Felker III
Handling mpeg correctly is already trivial, so we'll stick with avi if
that's ok...

Ok, but it is exactly the same thing. It would also solve the broken
frames after seeking for mpeg.
It is exactly the same problem.

IMO not at all. The problem with AVI is that you don't have the
information to reassign the frame numbers until you've already decoded
them, and then it's too late. With mpeg container, you already have
proper DTS and PTS, and any good codec will give you frames in the
right (display) order and with the correct timestamps.

- handling presentation order at application level
- using pts/dts
The 1st one is easier for avi, the second one for mpeg.
Using pts/dts is better as it ease sync with audio.

Maybe it's easier for an _editor_ which is free to be slow and reindex
the whole movie, but it's not possible for any kind of realtime
processing. In any case, I'm looking for a way to get a correct
display-order frame number for a keyframe after seeking to it, nothing
more. If you have any ideas for how to do this I'd be happy to hear
them! Having the frames in the right order without delay (or with
constant delay) is not enough; the design i'm working on depends on
always having absolutely correct timestamps for everything.

Post by Mean

Post by D Richard Felker III
As for the broken frames after seeking, it's just a matter of failing
to call the codec's flush() function.

yes, and handling the delay if you don"t have pts/dts.

With mpeg container, you _do_ have pts/dts, so no worry.

Post by Mean

Post by D Richard Felker III
BTW....what happens when you put xvid w/B-frames in .ogm? I hate to
think about it....

Same story, you reorder but first create an index.
Ogm is fine to play sequentially, horrible to edit.

Well ogm has timestamps, but they're sometimes broken. Are they
accurate with B frames? If so, it's trivial, just like mpeg. If not,
it's hell like avi (possibly even moreso).

Rich



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Mean]

Post by D Richard Felker III
Maybe it's easier for an _editor_ which is free to be slow and reindex
the whole movie, but it's not possible for any kind of realtime
processing.

Yes, that is true. In my case a long time at startup is acceptable,
not in yours.
(there is no penalty afterward)

Post by D Richard Felker III
In any case, I'm looking for a way to get a correct
display-order frame number for a keyframe after seeking to it, nothing
more. If you have any ideas for how to do this I'd be happy to hear
them! Having the frames in the right order without delay (or with
constant delay) is not enough; the design i'm working on depends on
always having absolutely correct timestamps for everything.

Is decoding ahead of displaying acceptable ?

If you decode/buffer X frames before displaying them (X >=3) :
- if there is no B frame in buffers, take the 1st frames as synchro
start point.
- if B frames are present, discard the oldest frame and resync on the
1st I frame found.
(it will absorb also the B frames from previous gop if any)

It is compatible with PTS/DTS frame and divx packed frames.

It is sub optimal (small delay + buffering) but relatively easy
as after that you are back to monotonous behaviour (one in, one out)

There will be an impact on performance though due to buffering.







-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[D Richard Felker III]

Post by Mean

Post by D Richard Felker III
In any case, I'm looking for a way to get a correct
display-order frame number for a keyframe after seeking to it, nothing
more. If you have any ideas for how to do this I'd be happy to hear
them! Having the frames in the right order without delay (or with
constant delay) is not enough; the design i'm working on depends on
always having absolutely correct timestamps for everything.

Is decoding ahead of displaying acceptable ?

Only to a certain extent. We might be direct rendering into video
memory, in which case we're limited to a small number of buffers. In
practice, using more than 2 buffers ahead will probably hurt
performance.

If the hurry_up flag is set, the rogue B frames from before the I
frame will get dropped but reported by lavc, right? This might allow
us to implement your idea without wasting buffers to decode the B
frames. Otherwise, some hack could be put into place to keep direct
rendering from being used for the B frames.

Post by Mean
- if there is no B frame in buffers, take the 1st frames as synchro
start point.
- if B frames are present, discard the oldest frame and resync on the
1st I frame found.
(it will absorb also the B frames from previous gop if any)

What do you mean by resync? Output the I frame using the dts from the
last dropped B frame as the pts?

Post by Mean
It is compatible with PTS/DTS frame and divx packed frames.

Yes. :)

Post by Mean
It is sub optimal (small delay + buffering) but relatively easy
as after that you are back to monotonous behaviour (one in, one out)

The one-in, one-out doesn't help us at all, since the whole core is
driven from the output end rather than the input (it pulls frames from
the decoder rather than pushing packets in from the demuxer).

Post by Mean
There will be an impact on performance though due to buffering.

Yes. :((

Rich



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Mean]

Post by D Richard Felker III

Post by Mean
Is decoding ahead of displaying acceptable ?

Only to a certain extent. We might be direct rendering into video
memory, in which case we're limited to a small number of buffers. In
practice, using more than 2 buffers ahead will probably hurt
performance.
If the hurry_up flag is set, the rogue B frames from before the I
frame will get dropped but reported by lavc, right? This might allow
us to implement your idea without wasting buffers to decode the B
frames. Otherwise, some hack could be put into place to keep direct
rendering from being used for the B frames.

Yes, it seems good. The only requisit is to know if the stream contains
B-frame at all or not to adapt the strategy.
If no B frame, take the 1st one else ignore it and skip B frames
with hurry up.

Concerning the frame type correctly reported if hurry up is set,
i know it does not work if hurry_up=5, maybe it is ok for lesser value.

In any case, it is simple to do it.

Post by D Richard Felker III

Post by Mean
- if there is no B frame in buffers, take the 1st frames as synchro
start point.
- if B frames are present, discard the oldest frame and resync on the
1st I frame found.
(it will absorb also the B frames from previous gop if any)

What do you mean by resync? Output the I frame using the dts from the
last dropped B frame as the pts?

Yes, i mean you know the PTS of the I frame you seek so you can
start your pts clock from that frame.

Post by D Richard Felker III

Post by Mean
It is compatible with PTS/DTS frame and divx packed frames.

Yes. :)

Post by Mean
It is sub optimal (small delay + buffering) but relatively easy
as after that you are back to monotonous behaviour (one in, one out)

The one-in, one-out doesn't help us at all, since the whole core is
driven from the output end rather than the input (it pulls frames from
the decoder rather than pushing packets in from the demuxer).

I mean it does not cause huge cpu consumption that may affect a/v sync.
In the method i use, it causes cpu peak followed by 0 cpu activity.

Anyway, if buffering some frames is acceptable, i think it will work :)



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[D Richard Felker III]

Post by Mean

Post by D Richard Felker III
Only to a certain extent. We might be direct rendering into video
memory, in which case we're limited to a small number of buffers. In
practice, using more than 2 buffers ahead will probably hurt
performance.
If the hurry_up flag is set, the rogue B frames from before the I
frame will get dropped but reported by lavc, right? This might allow
us to implement your idea without wasting buffers to decode the B
frames. Otherwise, some hack could be put into place to keep direct
rendering from being used for the B frames.

Yes, it seems good. The only requisit is to know if the stream contains
B-frame at all or not to adapt the strategy.

IMO this should be done whenever low_delay==0 in the codec context. It
won't hurt if there are no B frames to skip...

Post by Mean
If no B frame, take the 1st one else ignore it and skip B frames
with hurry up.
Concerning the frame type correctly reported if hurry up is set,
i know it does not work if hurry_up=5, maybe it is ok for lesser value.

No problem: hurry_up=5 would be disastrous here, since it would skip
I/P frames too!! I'm pretty sure hurry_up=2 (drop B frames only) sets
the frame type.

Rich



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Mean]

Post by D Richard Felker III

Post by Mean

Post by D Richard Felker III
If in addition to this, the first reference frame after the seek (the
I-frame) is not decoded until the second reference frame is found,
then it becomes possible to obtain an accurate frame number in avi
files! The (display-order) frame number of the I-frame you seek to is
exactly the (coded-order) frame number of the frame that causes it to
be decoded, minus 1.

There is a bit more to that.
Depending if B frames are present or not (low delay flag in header), the
codec (lavcodec or
mpeg2dec) will hold the frames for possibly 2 ticks before sending them out
to guarantee one frame in, one frame out and avoid possible variable
latency.
There is another case that breaks the rule, divx (or xvid) PB frames that
packs the B frame with its refernce frame and insert a null frame afterward
xxx I B P B PB xxxx
=> xx I (BP) 0 (BP) 0 xxxx
That forces the decision to be made at codec level and cause a pseudo
low delay.
That is B fame but with low delay.
you need to decode the null frame as it contains the second part of the
PB frame.
- low delay or PB frames : 0 delay
- real B frames : 2 frames delay
That makes frame accuracy / timestamp difficult

Is there any way to distinguish the two types? Could lavc somehow
provide this info to the caller?

Just add this in h263dec.c and you will know (have to decode a couple of
frame to get the info)

int av_is_voppacked(AVCodecContext *avctx)
{
MpegEncContext *s = avctx->priv_data;
if(s->divx_packed) return 1;
return 0;

}

Post by D Richard Felker III
Does the divx/xvid method allow more than one B frame?
i.e. I (BBP) 0 0 (BPP) 0 0 ...?

I don"t know, older divx version did not allow more than one B frame.
I have not looked since.

Post by D Richard Felker III
As stupid as it is, I'm inclined to think that the divx/xvid method is
the only possible correct way of storing video with B frames in an avi
file, and mencoder is stupid for storing the frames the way it does...
But, in any case, I want MPlayer G2 to be able to handle _all_ files
with 100% exact pts, so some workaround needs to be found...

Yes and no
As far as playback is concerned, PB frame are nice, it hides B frame
problem to applications.

For editing, it is a pain as you don"t really master how the codec will
handle this
and have to double guessing.

I prefer the mpeg way (DTS/PTS order) that enable you to have the same
core code
to handle both cases. The container (avi/raw mpeg/vob/...) does not
matter much then.

If you use PB frames for avi, and you cannot use them for mpeg you have
to use
2 different strategies to sync/timestamp.







-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[D Richard Felker III]

Post by Mean

Post by D Richard Felker III

Post by Mean
- low delay or PB frames : 0 delay
- real B frames : 2 frames delay
That makes frame accuracy / timestamp difficult

Is there any way to distinguish the two types? Could lavc somehow
provide this info to the caller?

Just add this in h263dec.c and you will know (have to decode a couple of
frame to get the info)
int av_is_voppacked(AVCodecContext *avctx)
{
MpegEncContext *s = avctx->priv_data;
if(s->divx_packed) return 1;
return 0;
}

Moving the flag to AVCodecContext would probably make more sense... :)

Post by Mean

Post by D Richard Felker III
As stupid as it is, I'm inclined to think that the divx/xvid method is
the only possible correct way of storing video with B frames in an avi

Yes and no
As far as playback is concerned, PB frame are nice, it hides B frame
problem to applications.

Yes.

Post by Mean
For editing, it is a pain as you don"t really master how the codec will
handle this
and have to double guessing.

I don't think these files are really intended for editing, with gop
sizes as large as 300... You're right though -- the null frames are
really a pain.

Post by Mean
I prefer the mpeg way (DTS/PTS order) that enable you to have the same
core code
to handle both cases. The container (avi/raw mpeg/vob/...) does not
matter much then.

Actually you have trouble with mpeg2-in-avi too. (In fact, it was one
of my test cases.) Ultimately pts comes from the container, and if you
have no way to get accurate pts from it, you're out of luck.

Post by Mean
If you use PB frames for avi, and you cannot use them for mpeg you have
to use
2 different strategies to sync/timestamp.

Yes, we have a "broken fileformat" flag for avi demuxer to use that
says B frame pts is bogus... :)

Rich



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Edouard Gomez]

Post by D Richard Felker III
Does the divx/xvid method allow more than one B frame?
i.e. I (BBP) 0 0 (BPP) 0 0 ...?

It doesn't pack more bframes. To have 0 delay bframe decoding we only
need to have the first bframe + the second reference packed together,
all other bframes, if there are more can be written alone as "usual"
after the 0 frame.

Stream ordered, that gives:
I (PB) 0 B* (PB)

Be aware, that divx did use a trick to inform their decoder of packed
data (the p in the VOL user data divx signature) and that the 0 frame
isn't really a 0 frame but a frame whose timestamp is faked... I'm
probably not very precise, that is a long time we don't touch at that
code in XviD.

For more details see the file src/encoder.c around L1014 in a 1.0.0
beta release.

[D Richard Felker III]

Post by Edouard Gomez

Post by D Richard Felker III
Does the divx/xvid method allow more than one B frame?
i.e. I (BBP) 0 0 (BPP) 0 0 ...?

It doesn't pack more bframes. To have 0 delay bframe decoding we only
need to have the first bframe + the second reference packed together,
all other bframes, if there are more can be written alone as "usual"
after the 0 frame.
I (PB) 0 B* (PB)

Ah, makes sense.

Post by Edouard Gomez
Be aware, that divx did use a trick to inform their decoder of packed
data (the p in the VOL user data divx signature) and that the 0 frame
isn't really a 0 frame but a frame whose timestamp is faked... I'm

There are no such things as timestamps in AVI files. That's the whole
problem... :/

Rich



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Edouard Gomez]

Post by D Richard Felker III
There are no such things as timestamps in AVI files. That's the whole
problem... :/

I meant inside the mpeg4 stream.

[Michael Niedermayer]

Hi

Post by D Richard Felker III

Post by Mean

Post by D Richard Felker III
If in addition to this, the first reference frame after the seek (the
I-frame) is not decoded until the second reference frame is found,
then it becomes possible to obtain an accurate frame number in avi
files! The (display-order) frame number of the I-frame you seek to is
exactly the (coded-order) frame number of the frame that causes it to
be decoded, minus 1.

There is a bit more to that.
Depending if B frames are present or not (low delay flag in header), the
codec (lavcodec or
mpeg2dec) will hold the frames for possibly 2 ticks before sending them
out to guarantee one frame in, one frame out and avoid possible variable
latency.

!?
b frames will be output immedeatly, IP has variable delay if low_delay=0

Post by D Richard Felker III

Post by Mean
There is another case that breaks the rule, divx (or xvid) PB frames that
packs the B frame with its refernce frame and insert a null frame afterward
xxx I B P B PB xxxx
=> xx I (BP) 0 (BP) 0 xxxx
That forces the decision to be made at codec level and cause a pseudo
low delay.
That is B fame but with low delay.
you need to decode the null frame as it contains the second part of the
PB frame.

huh? the null frame is just a dummy, theres nothing to decode, and the P frame
is before the B so its I PB PB ... not BP BP

Post by D Richard Felker III

Post by Mean
- low delay or PB frames : 0 delay
- real B frames : 2 frames delay
That makes frame accuracy / timestamp difficult

Is there any way to distinguish the two types? Could lavc somehow
provide this info to the caller?

theres a 'p' in the userdata for the PB packed style IIRC, we should export
that

Post by D Richard Felker III
Does the divx/xvid method allow more than one B frame?
i.e. I (BBP) 0 0 (BPP) 0 0 ...?

AFAIK no, and i also would like to mention that the packed PB frame style is
not legel mpeg4

Post by D Richard Felker III
As stupid as it is, I'm inclined to think that the divx/xvid method is
the only possible correct way of storing video with B frames in an avi
file, and mencoder is stupid for storing the frames the way it does...
But, in any case, I want MPlayer G2 to be able to handle _all_ files
with 100% exact pts, so some workaround needs to be found...

there are some timetamps in the mpeg4 headers which may be usefull, they also
must be set for direct mode prediction in b frames to work, so with some luck
it may be possible to use them ...

[...]
--
Michael
level[i]= get_vlc(); i+=get_vlc(); (violates patent EP0266049)
median(mv[y-1][x], mv[y][x-1], mv[y+1][x+1]); (violates patent #5,905,535)
buf[i]= qp - buf[i-1]; (violates patent #?)
for more examples, see http://mplayerhq.hu/~michael/patent.html
stop it, see http://petition.eurolinux.org & http://petition.ffii.org/eubsa/en


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Mean]

Post by Michael Niedermayer
Hi

Post by Mean

Post by D Richard Felker III
If in addition to this, the first reference frame after the seek (the
I-frame) is not decoded until the second reference frame is found,
then it becomes possible to obtain an accurate frame number in avi
files! The (display-order) frame number of the I-frame you seek to is
exactly the (coded-order) frame number of the frame that causes it to
be decoded, minus 1.

There is a bit more to that.
Depending if B frames are present or not (low delay flag in header), the
codec (lavcodec or
mpeg2dec) will hold the frames for possibly 2 ticks before sending them
out to guarantee one frame in, one frame out and avoid possible variable
latency.

!?
b frames will be output immedeatly, IP has variable delay if low_delay=0

If we do dont force low delay and if you have this as incoming
I P B B
You will have this as out
NULL NULL I B B P

It is a global delay of 2 frames, as the 1st B frame cannot be
decoded before 3 frames has been fed.

Post by Michael Niedermayer

Post by Mean
you need to decode the null frame as it contains the second part of the
PB frame.

huh? the null frame is just a dummy, theres nothing to decode, and the P frame
is before the B so its I PB PB ... not BP BP

Normally when a null frame is met, it does not even reach the codec as
it means
in avi language dropped frame/repeat previous but it means nothing for a
codec.

And some codecs do not like being asked to decode a zero sized frame.

In that case, you have to call the codec anyway as it knows it will use
the 2nd frame
in the packed previous frame.



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Michael Niedermayer]

Hi

Post by Mean

Post by Michael Niedermayer
Hi

Post by Mean

Post by D Richard Felker III
If in addition to this, the first reference frame after the seek (the
I-frame) is not decoded until the second reference frame is found,
then it becomes possible to obtain an accurate frame number in avi
files! The (display-order) frame number of the I-frame you seek to is
exactly the (coded-order) frame number of the frame that causes it to
be decoded, minus 1.

There is a bit more to that.
Depending if B frames are present or not (low delay flag in header), the
codec (lavcodec or
mpeg2dec) will hold the frames for possibly 2 ticks before sending them
out to guarantee one frame in, one frame out and avoid possible variable
latency.

!?
b frames will be output immedeatly, IP has variable delay if low_delay=0

If we do dont force low delay and if you have this as incoming
I P B B
You will have this as out
NULL NULL I B B P
It is a global delay of 2 frames, as the 1st B frame cannot be
decoded before 3 frames has been fed.

no
encoder in: IBBPBBP
bitstreeam: xxIPBBPBB
decoder out: xxxIBBPBBP
the delay fom encoding->bitstream is variable, same for bitstream->decoding,
and their sum is a delay of 3 not 2 in the case of 2 b frames

Post by Mean

Post by Michael Niedermayer

Post by Mean
you need to decode the null frame as it contains the second part of the
PB frame.

huh? the null frame is just a dummy, theres nothing to decode, and the P
frame is before the B so its I PB PB ... not BP BP

Normally when a null frame is met, it does not even reach the codec as
it means
in avi language dropped frame/repeat previous but it means nothing for a
codec.

no, the null frames which are used in packed divx style PB frames have nothing
to do with skiped frames in AVI

Post by Mean
And some codecs do not like being asked to decode a zero sized frame.

they are not zero sized, they contain a complete mpeg4 VOP header

[...]
--
Michael
level[i]= get_vlc(); i+=get_vlc(); (violates patent EP0266049)
median(mv[y-1][x], mv[y][x-1], mv[y+1][x+1]); (violates patent #5,905,535)
buf[i]= qp - buf[i-1]; (violates patent #?)
for more examples, see http://mplayerhq.hu/~michael/patent.html
stop it, see http://petition.eurolinux.org & http://petition.ffii.org/eubsa/en


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Mean]

Post by Michael Niedermayer

Post by Mean

Post by Michael Niedermayer
!?
b frames will be output immedeatly, IP has variable delay if low_delay=0

If we do dont force low delay and if you have this as incoming
I P B B
You will have this as out
NULL NULL I B B P
It is a global delay of 2 frames, as the 1st B frame cannot be
decoded before 3 frames has been fed.

no
encoder in: IBBPBBP
bitstreeam: xxIPBBPBB
decoder out: xxxIBBPBBP
the delay fom encoding->bitstream is variable, same for bitstream->decoding,
and their sum is a delay of 3 not 2 in the case of 2 b frames

We were talking of decoding only, so bitstream -> decoder out
The problem is that there is a delay between in and out caused by
B-frame,
the more numerical part was from memory (specifically libmpeg2) but
i'm probably wrong as i always work with low_delay anyway.

Post by Michael Niedermayer

Post by Mean

Post by Michael Niedermayer

Post by Mean
you need to decode the null frame as it contains the second part of the
PB frame.

huh? the null frame is just a dummy, theres nothing to decode, and the P
frame is before the B so its I PB PB ... not BP BP

Normally when a null frame is met, it does not even reach the codec as
it means
in avi language dropped frame/repeat previous but it means nothing for a
codec.

no, the null frames which are used in packed divx style PB frames have nothing
to do with skiped frames in AVI

Post by Mean
And some codecs do not like being asked to decode a zero sized frame.

they are not zero sized, they contain a complete mpeg4 VOP header

Yes, you are right., they are not 0 sized.




-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[D Richard Felker III]

Post by Michael Niedermayer

Post by Mean
If we do dont force low delay and if you have this as incoming
I P B B
You will have this as out
NULL NULL I B B P
It is a global delay of 2 frames, as the 1st B frame cannot be
decoded before 3 frames has been fed.

no
encoder in: IBBPBBP
bitstreeam: xxIPBBPBB
decoder out: xxxIBBPBBP
the delay fom encoding->bitstream is variable, same for bitstream->decoding,
and their sum is a delay of 3 not 2 in the case of 2 b frames

Yes, this makes more sense.

Post by Michael Niedermayer

Post by Mean
Normally when a null frame is met, it does not even reach the
codec as it means in avi language dropped frame/repeat previous
but it means nothing for a codec.

no, the null frames which are used in packed divx style PB frames have nothing
to do with skiped frames in AVI

Post by Mean
And some codecs do not like being asked to decode a zero sized frame.

they are not zero sized, they contain a complete mpeg4 VOP header

Very good to know! Thanks!

Rich



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[D Richard Felker III]

Post by Mean
There is another case that breaks the rule, divx (or xvid) PB frames that
packs the B frame with its refernce frame and insert a null frame afterward
xxx I B P B PB xxxx
=> xx I (BP) 0 (BP) 0 xxxx
That forces the decision to be made at codec level and cause a pseudo
low delay.
That is B fame but with low delay.

I decided today to check and see if lavc would apply delay on such
files or not... But what I found was quite surprising! The only
windows-made xvid avi file I could find does NOT use this silly
packing, but instead is stored like what mencoder would do!

Example at:

http://brightrain.aerifal.cx/~dalias/video_examples/avi-bframes/

So, as for my original question, what will lavc do with "packed" b
frames files? Could anyone provide me with a sample of one?

Rich



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Michael Niedermayer]

Hi

Post by D Richard Felker III

Post by Mean
There is another case that breaks the rule, divx (or xvid) PB frames that
packs the B frame with its refernce frame and insert a null frame afterward
xxx I B P B PB xxxx
=> xx I (BP) 0 (BP) 0 xxxx
That forces the decision to be made at codec level and cause a pseudo
low delay.
That is B fame but with low delay.

I decided today to check and see if lavc would apply delay on such
files or not... But what I found was quite surprising! The only
windows-made xvid avi file I could find does NOT use this silly
packing, but instead is stored like what mencoder would do!
http://brightrain.aerifal.cx/~dalias/video_examples/avi-bframes/
So, as for my original question, what will lavc do with "packed" b
frames files?

delay decoding of the b frame, so its decoded during the null frame

Post by D Richard Felker III
Could anyone provide me with a sample of one?

http://www.mplayerhq.hu/~michael/m.avi

[...]
--
Michael
level[i]= get_vlc(); i+=get_vlc(); (violates patent EP0266049)
median(mv[y-1][x], mv[y][x-1], mv[y+1][x+1]); (violates patent #5,905,535)
buf[i]= qp - buf[i-1]; (violates patent #?)
for more examples, see http://mplayerhq.hu/~michael/patent.html
stop it, see http://petition.eurolinux.org & http://petition.ffii.org/eubsa/en


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[D Richard Felker III]

Post by Michael Niedermayer
Hi

Post by D Richard Felker III

Post by Mean
There is another case that breaks the rule, divx (or xvid) PB frames that
packs the B frame with its refernce frame and insert a null frame afterward
xxx I B P B PB xxxx
=> xx I (BP) 0 (BP) 0 xxxx
That forces the decision to be made at codec level and cause a pseudo
low delay.
That is B fame but with low delay.

I decided today to check and see if lavc would apply delay on such
files or not... But what I found was quite surprising! The only
windows-made xvid avi file I could find does NOT use this silly
packing, but instead is stored like what mencoder would do!
http://brightrain.aerifal.cx/~dalias/video_examples/avi-bframes/
So, as for my original question, what will lavc do with "packed" b
frames files?

delay decoding of the b frame, so its decoded during the null frame

So then the display-order frame number (pts) is the same as the avi
frame number that you just passed to the decoder?

IMO this is very nice, but it's also a problem, since the program
using libavcodec can't tell whether B frames are packed or not (and
thus can't tell whether to compensate for delay or not!).

Post by Michael Niedermayer

Post by D Richard Felker III
Could anyone provide me with a sample of one?

http://www.mplayerhq.hu/~michael/m.avi

Thanks, I'll take a look.

Rich



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Michael Niedermayer]

Hi

Post by D Richard Felker III

Post by Michael Niedermayer
Hi

Post by D Richard Felker III

Post by Mean
There is another case that breaks the rule, divx (or xvid) PB frames
that packs the B frame with its refernce frame and insert a null
frame afterward
xxx I B P B PB xxxx
=> xx I (BP) 0 (BP) 0 xxxx
That forces the decision to be made at codec level and cause a pseudo
low delay.
That is B fame but with low delay.

I decided today to check and see if lavc would apply delay on such
files or not... But what I found was quite surprising! The only
windows-made xvid avi file I could find does NOT use this silly
packing, but instead is stored like what mencoder would do!
http://brightrain.aerifal.cx/~dalias/video_examples/avi-bframes/
So, as for my original question, what will lavc do with "packed" b
frames files?

delay decoding of the b frame, so its decoded during the null frame

So then the display-order frame number (pts) is the same as the avi
frame number that you just passed to the decoder?

i dont fully understand this question, ill try after a few hours of sleep
again :)

Post by D Richard Felker III
IMO this is very nice, but it's also a problem, since the program
using libavcodec can't tell whether B frames are packed or not (and
thus can't tell whether to compensate for delay or not!).

hmm, theres AVCodecContext.delay, its only used for encoding currently but we
could use it here too (1 for PB style, 0 otherwise)

[...]
--
Michael
level[i]= get_vlc(); i+=get_vlc(); (violates patent EP0266049)
median(mv[y-1][x], mv[y][x-1], mv[y+1][x+1]); (violates patent #5,905,535)
buf[i]= qp - buf[i-1]; (violates patent #?)
for more examples, see http://mplayerhq.hu/~michael/patent.html
stop it, see http://petition.eurolinux.org & http://petition.ffii.org/eubsa/en


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[D Richard Felker III]

Post by Michael Niedermayer

Post by D Richard Felker III

Post by Michael Niedermayer

Post by D Richard Felker III
So, as for my original question, what will lavc do with "packed" b
frames files?

delay decoding of the b frame, so its decoded during the null frame

So then the display-order frame number (pts) is the same as the avi
frame number that you just passed to the decoder?

i dont fully understand this question, ill try after a few hours of sleep
again :)

OK, let me explain better too:

First case, you have a non-packed avi file, and the following frames
(display order):

I0 B1 B2 P3 B4 B5 P6 B7 B8 I9 ...
0 1 2 3 4 5 6 7 8 9 <-- display frame number

In the avi container, they're stored as:

I0 P3 B1 B2 P6 B4 B5 I9 B7 B8 ...
0 1 2 3 4 5 6 7 8 9 <-- input (avi) frame number

And lavc's output when decoding is:

-- I0 B1 B2 P3 B4 B5 P6 B7 B8 I9 ...
0 1 2 3 4 5 6 7 8 9 10 <-- decoding frame number
(i.e. input packet # that caused
the frame to be output by lavc)

Thus when you decode, the input frame number (from the avi) doesn't in
general correspond to the display-order frame number. In fact the only
good situation is when you have a B frame, in which case the input
(avi) frame number and decoding frame number are equal, and are both
one plus the display frame number. In general you have a mess.

Now let's look at the packed case:

B1 B4 B7
I0 P3 B2 -- P6 B5 -- I9 B8 -- ...
0 1 2 3 4 5 6 7 8 9

If I understand right, lavc's output order is:

I0 B1 B2 P3 B4 B5 P6 B7 B8 I9 ...
0 1 2 3 4 5 6 7 8 9

i.e. exactly the same as if the avi file didn't have B frames. Since
the avi container is too primitive to have any pts information, and
since trying to store B frames non-packed makes determining pts at
decode-time _very_ painful (it requires careful interplay between
demuxer and decoder), I think the packed system makes a lot more
sense.

Now, the questions:

1. Am I right about lavc's decoding order in the packed case?

and...

Post by Michael Niedermayer

Post by D Richard Felker III
IMO this is very nice, but it's also a problem, since the program
using libavcodec can't tell whether B frames are packed or not (and
thus can't tell whether to compensate for delay or not!).

hmm, theres AVCodecContext.delay, its only used for encoding currently but we
could use it here too (1 for PB style, 0 otherwise)

2. How can the program using lavc determine whether there's a delay or
not? AFAICT, with the current lavc behavior, you _must_ output
wrong pts in one case or the other... :(

Rich




-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Michael Niedermayer]

Hi

Post by D Richard Felker III

Post by Michael Niedermayer

Post by D Richard Felker III

Post by Michael Niedermayer

Post by D Richard Felker III
So, as for my original question, what will lavc do with "packed" b
frames files?

delay decoding of the b frame, so its decoded during the null frame

So then the display-order frame number (pts) is the same as the avi
frame number that you just passed to the decoder?

i dont fully understand this question, ill try after a few hours of sleep
again :)

First case, you have a non-packed avi file, and the following frames
I0 B1 B2 P3 B4 B5 P6 B7 B8 I9 ...
0 1 2 3 4 5 6 7 8 9 <-- display frame number
I0 P3 B1 B2 P6 B4 B5 I9 B7 B8 ...
0 1 2 3 4 5 6 7 8 9 <-- input (avi) frame number
-- I0 B1 B2 P3 B4 B5 P6 B7 B8 I9 ...
0 1 2 3 4 5 6 7 8 9 10 <-- decoding frame number
(i.e. input packet # that caused
the frame to be output by lavc)
Thus when you decode, the input frame number (from the avi) doesn't in
general correspond to the display-order frame number. In fact the only
good situation is when you have a B frame, in which case the input
(avi) frame number and decoding frame number are equal, and are both
one plus the display frame number. In general you have a mess.
B1 B4 B7
I0 P3 B2 -- P6 B5 -- I9 B8 -- ...
0 1 2 3 4 5 6 7 8 9
I0 B1 B2 P3 B4 B5 P6 B7 B8 I9 ...
0 1 2 3 4 5 6 7 8 9
i.e. exactly the same as if the avi file didn't have B frames. Since
the avi container is too primitive to have any pts information, and
since trying to store B frames non-packed makes determining pts at
decode-time _very_ painful (it requires careful interplay between
demuxer and decoder), I think the packed system makes a lot more
sense.
1. Am I right about lavc's decoding order in the packed case?

no, this is the way divx decodes it, but theres a very serious problem with
this, as u must decode 2 frames at once which for single thread players with
few buffers simply means that twice as much cpu time is needed for equally
smooth playback
so lavc simply handles the stream as if the b frames where stored in the null
frames

Post by D Richard Felker III
-- I0 B1 B2 P3 B4 B5 P6 B7 B8 I9 ...
0 1 2 3 4 5 6 7 8 9 10 <-- decoding frame number

[...]
--
Michael
level[i]= get_vlc(); i+=get_vlc(); (violates patent EP0266049)
median(mv[y-1][x], mv[y][x-1], mv[y+1][x+1]); (violates patent #5,905,535)
buf[i]= qp - buf[i-1]; (violates patent #?)
for more examples, see http://mplayerhq.hu/~michael/patent.html
stop it, see http://petition.eurolinux.org & http://petition.ffii.org/eubsa/en


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[D Richard Felker III]

Post by Michael Niedermayer

Post by D Richard Felker III
1. Am I right about lavc's decoding order in the packed case?

no, this is the way divx decodes it, but theres a very serious problem with
this, as u must decode 2 frames at once which for single thread players with
few buffers simply means that twice as much cpu time is needed for equally
smooth playback
so lavc simply handles the stream as if the b frames where stored in the null
frames

Post by D Richard Felker III
-- I0 B1 B2 P3 B4 B5 P6 B7 B8 I9 ...
0 1 2 3 4 5 6 7 8 9 10 <-- decoding frame number

OK, then I guess it's not as critical to know which case we're dealing
with then. On the other hand, it would still be nice to know if we
have a packed-divx avi, since it would let us assign timestamps
directly to all frames rather than just the B frames.

Rich



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Michael Niedermayer]

Hi

Post by D Richard Felker III
I'm working on mplayer g2, and trying to work out the horrible
nightmare of pts with B-frames in avi files. From what I can gather,
B-frames in avi files seem to be stored in codec order, meaning that
their frame numbers within the avi file do _not_ represent their
display order. This is very troubling, since it makes it nearly
impossible to get a correct timestamp after seeking!
So far, I've only come across one way to accurately determine frame
numbers in display order: the display-order frame number of a
reference frame (I/P) is the coded-order frame number of the next
reference frame, minus 1. However, I'm concerned that there's no way
to see where the next reference frame is before lavc decodes the
current one.
There's a related problem with B-frames in MPlayer, the notorious
glitch-after-seeking bug. In case you're not familiar with it, MPlayer
will show a bogus blocky mix of the scene before the seek and the new
scene after the seek for a few frames after seeking, when playing
Suppose after seeking, the next 4 frames (coded order) are IBBP. The
two B-frames actually belong before the I-frame, and should not be
shown after seeking. But the decoder (lavc or libmpeg2) has no way of
knowing this, and thinks the two B-frames belong between the pre-seek
reference frame and the new I-frame. Thus, broken output.
The only real fix I can think of for this problem, is to provide a
method for the caller to notify lavc that a seek has taken place, and
to "flush the buffers". After such a notification, all B-frames should
be dropped until the second reference frame is found.

AVCodec.flush() !

[...]
--
Michael
level[i]= get_vlc(); i+=get_vlc(); (violates patent EP0266049)
median(mv[y-1][x], mv[y][x-1], mv[y+1][x+1]); (violates patent #5,905,535)
buf[i]= qp - buf[i-1]; (violates patent #?)
for more examples, see http://mplayerhq.hu/~michael/patent.html
stop it, see http://petition.eurolinux.org & http://petition.ffii.org/eubsa/en


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[D Richard Felker III]

Post by Michael Niedermayer

Post by D Richard Felker III
The only real fix I can think of for this problem, is to provide a
method for the caller to notify lavc that a seek has taken place, and
to "flush the buffers". After such a notification, all B-frames should
be dropped until the second reference frame is found.

AVCodec.flush() !

Will the bogus B-frames be dropped if we call AVCodec.flush()?

Rich



-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click

[Michael Niedermayer]

Hi

Post by D Richard Felker III

Post by Michael Niedermayer

Post by D Richard Felker III
The only real fix I can think of for this problem, is to provide a
method for the caller to notify lavc that a seek has taken place, and
to "flush the buffers". After such a notification, all B-frames should
be dropped until the second reference frame is found.

AVCodec.flush() !

Will the bogus B-frames be dropped if we call AVCodec.flush()?

yes, unless its buggy

[...]
--
Michael
level[i]= get_vlc(); i+=get_vlc(); (violates patent EP0266049)
median(mv[y-1][x], mv[y][x-1], mv[y+1][x+1]); (violates patent #5,905,535)
buf[i]= qp - buf[i-1]; (violates patent #?)
for more examples, see http://mplayerhq.hu/~michael/patent.html
stop it, see http://petition.eurolinux.org & http://petition.ffii.org/eubsa/en


-------------------------------------------------------
This SF.net email is sponsored by: IBM Linux Tutorials.
Become an expert in LINUX or just sharpen your skills. Sign up for IBM's
Free Linux Tutorials. Learn everything from the bash shell to sys admin.
Click now! http://ads.osdn.com/?ad_id=1278&alloc_id=3371&op=click