jsmar

jsmar

It's possible that that limit existed in the Harris MPH submittal before it was merged to create the ATSC M/H standard.
Yeah, it takes the ninth slot to be allocated before a slot in the lower half of a VSB field gets used (slots are allocated 1 in 4, then 1 in 2 before going to every slot). Since the ATSC randomizer is synced to the beginning of the field, I have to adjust for such a case. But I suspect I'll never actually get to test that situation.
I was going to send you an update via email, but I thought perhaps this may be of interest to others in this forum. I found and joined this forum because there didn't seem to be an equivalent forum on the AVS forum. I'm not sure the level of detail that will follow will be of interest to others in this forum, but I figured I'd do it once, and let people tell me if this is at an inappropriate level.

Yes, I've already kind of mentioned that I was going to try to decode mobile DTV data elsewhere, and I inferred it above. I've already started writing code, because there is no better way to learn a standard than to actually try to implement it. You can read the same thing over and over again, but not really understand what it is saying. When you try to implement it you also find the holes (i.e. the stuff that is not mentioned that may require reading between the lines).

I assume you are referring to the ability to decode mobile DTV using a Legacy DTV decoder chip that has no knowledge of the Mobile DTV standard. Sure, that does interest me, but I'm not sure how useful this ability will be. My primary goal is being able to get detailed technical information out of the mobile stream, not to get video, although I may do something in that area to satisfy my curiosity.

First, you've been assuming that the mobile data obtained via a legacy decoder would actually be decodeable, whereas I have had my doubts. I noticed a few statements at other locations that seemed to indicate that this ability was actually a goal of the standard, but it clearly is not. The actual goal was just that mobile DTV data not interfere with the function of legacy receivers in any way.

I wasn't even sure the data would show up and not be dropped by a legacy decoder, but then you had TSReader output showing mobile DTV data with either pid 0x1eee or pid 0x1ff9. So that interested me, and as you know, I pursued getting a capture from someone else, since there are no stations in my area currently broadcasting mobile DTV. I was able to get some captures from the Washington D.C. area.

But again, just because the legacy decoders were providing data didn't mean that it would be useful or complete. As I read the standard in more detail I went back and forth regarding whether I thought it would be possible or not. Let me give some details why I thought that it may not be possible.

Mobile DTV data is not meant to go through the standard ATSC decoding process, because at the transmitting side, a Mobile DTV aware exciter splits the mobile data out and handles it differently. After the data traverses the Studio to Transmitter Link, mobile DTV data follows a different path than the standard TS main services. The first step in the normal TS main service chain is to randomize the data. Mobile DTV data bypasses this step, because it was randomized earlier in the process, and it doesn't want to harm the "all powerful" mobile training sequences.

The next step is that the data is Reed Solomon encoded. Normal TS main data goes through a normal encoding process where the 187 data bytes in each packet are appended with 20 bytes of parity data. Mobile data rearranges every packet in a group (i.e. a different pattern for each of the 118 packets in that group) and does a non standard (i.e. non systematic) Reed Solomon encoding where the 20 parity bytes are placed in different and not even consecutive locations in the packet. So the resulting 207 byte packet has real data after the 187th byte. Once again, its all about finding holes for the training sequences, and also not breaking legacy receivers.

Next we reach the ATSC interleaver. Both TS main and Mobile DTV data go through this, but the Mobile DTV data was "de-interleaved" in a complementary fashion earlier, so the interleaver is actually restoring the data to the order that a Mobile DTV decoder would like to see it.

The next stage is trellis encoding, and the Mobile DTV aware modified trellis encoder has the ability to support initialization sequences so that certain outputs can be guaranteed (once again, we gotta get those training sequences through unmolested). The remaining steps (pilot insertion, 8vsb modulation, etc. ) are common.

So, now let's look at how an ATSC legacy decoder is going to handle this. First, it appears that the trellis decoding is not an issue, i.e. the extra support for initialization doesn't affect the decoding side, so we are OK there. At this point a mobile DTV decoder would take the data as is and not follow the legacy path. The legacy decoder is going to de-interleave the data, but that is reversible, so we're OK there. It's the next step that really made me think all hope was lost for a while.

A legacy decoder thinks that every packet has 187 bytes of data, followed by 20 bytes of RS parity. So it error checks/corrects the data accordingly and then just DROPS the 20 bytes of what it thinks is RS parity. Oops. For Mobile DTV, those 20 bytes contain valid data. It wasn't even clear to me that the mobile RS encoder was following the same algorithm, since it was placing parity distributed throughout the packet. Luckily the "do no harm" goal bails us out here. At first I wasn't sure why it would make any difference if the legacy decoders detected a bunch of parity errors, since it wasn't going to decode those packets anyway. But then I realized that all those parity errors would affect the S/N quality meters on legacy tuners, making people think they were getting bad reception when they really were not, since only the M/H packets would appear to be bad. So, even though the parity was distributed throughout the packet, it had to satisfy the legacy decoders. Which means that those 20 bytes of valid data are recoverable by just reencoding the 187 bytes of data to generate the 20 legacy parity bytes, which are actually M/H data.

The next step in the decoding process is "de-randomizing". But note that the mobile DTV data was never randomized in the first place, so the legacy decoder is now randomizing it when it shouldn't. The randomizing process involves xor'ing the data with a pseudo random sequence that is synchronized with the beginning of a VSB frame. That's something that requires some information that the hardware decoder has (where the VSB frame starts) which is not available once the data is delivered from the decoder. Luckily each M/H slot can contain only 118 out of 156 packets as mobile data, and those 118 packets are consecutive. If all the packets could be mobile data then we'd receive a stream of mobile data and potentially have no idea where a slot starts. But given that there always has to be non mobile data between each 118 byte sequence, we can easily determine where packet 0 is. However, that is only half the problem, because packet 0 can be in the top half or the bottom half of a VSB frame, and the pseudo random sequence will be different for the two. As I mentioned above, we may never see mobile data from the bottom half in practice, but I have to allow for the possibility. In this case the known training data becomes useful. Packet 0 has 4 known training bytes at known locations within the packet. Since there are only two possible random sequences that can be used to encode the packet, I can simply try both and check the results against the known values. I can eventually extend this to extracting sync even when someone prefilters the transport stream and only provides the mobile DTV data. In that case I probably want to match more than 4 bytes, but some packets have 18-20 bytes of training data for me to compare against, which will allow me to detect and synchronize the stream.

In summary, I've only recently confirmed that it will be possible to decode M/H data from a legacy decoder. Note that I can't just take the data from the legacy decoder and continue decoding it. I first have to undo all the damage that the legacy decoder did in order to recover the M/H data before I can actually start to decode.

This all has some implications. First, hopefully noone expects to be able to receive M/H data with a legacy decoder while traveling. You will need a real M/H decoder to do that. If you can't receive the TS main services (i.e. the normal HD/SD channels being broadcast in the same data stream) then you're probably not going to be able to get the M/H data. Sure the M/H data has more error correcting code, but if the legacy decoder fails to recover the data using the standard RS code (which a mobile DTV decoder doesn't even use), you will not be able to recover the 20 data bytes that the legacy decoder dropped on the floor, probably making it almost impossible for those better RS codes to be effective. Besides, receiving data while moving is more about locking the signal in the first place, which is something a mobile DTV decoder does with the aid of those training sequences distributed throughout the group, which a legacy decoder knows nothing about. You can't recover what you can't receive in the first place.

Note also that there are three levels of mobile RS codes, and the top level, which adds 48 parity bytes, probably can't be decoded in realtime with the latest Pentium chips. I'm not going to even try. I'm going to leverage the only useful thing that the legacy decoder did for me, and that was recovering data via the legacy RS code (with 20 parity bytes effectively). I'm going to assume the data is valid beyond that point and just "turbo decode" the mobile RS coding by just dropping those 24/36/48 parity bytes. Perhaps I might be persuaded otherwise later if the "digital cliff" for this case is wider than I think, but it's a real low priority for me at this point. I'm certainly not going to write hand tuned assembly language to attempt real time decoding of this stuff.

Finally, even if I succeed in writing software that can decode this stuff, I'm not sure how useful it will be to others, or even if I will be allowed to distribute it. I work for a software company that "owns" my intellectual property, i.e. I can be fired for writing software and distributing it without the companies permission. So before I can provide anything I am going to need to get that permission. The process for releasing open source software is at least somewhat reasonable, but I can't say I'm looking forward to going through that.

Also, as I think I shared with Trip previously, I own an HD HomeRun and write all my software for Linux. I don't usually write software for Windows, although I have on occasion ported some command line stuff to Windows. So if you're hoping for Windows software you may be out of luck (although perhaps someone who cares enough can port whatever I produce if I get permission to distribute it).

The other issue is that there is no Video for Linux driver for the HDHomeRun, since it is a network tuner and not a USB/PCI tuner. So most of the software I've written is only useful for HDHomeRun owners, since I program it using Silicon Dust's proprietary API. I can probably port the software to a Video for Linux interface, but I would need access to a Video for Linux compatible tuner. Perhaps Trip's loaner program might be useful here . The software will be able to process raw TS files, so although cumbersome, you could capture (either on Windows or Linux) and then post process on Linux with my software (at least that is a goal).

Note that I also have no intention of actually writing video/audio decoders or incorporating such libraries in my software. If I even go as far in the decoding process to actually extract video/audio data (which is not a high priority for me, although I am certainly curious about the actual content) I intend to just extract the services and either save them in a file in a format that some media player could use, or stream it directly to a compatible media player (VLC for example).

So, the current status of my software is at the point of having restored the M/H data. It syncs and de-randomizes the data, then RS encodes the data to recover the missing 20 bytes, and then re-interleaves it so that it can now be decoded. Luckily I've been able to verify that the software is working correctly due to the known training sequences, some of which go through the 20 "parity" bytes, so I know that I am properly recovering real data and my understanding of what is going on matches the standard.

So, anyone other than Trip actually interested in this level of detail? What is the interest level in being able to extract mobile DTV programming with your non moving Linux desktop/laptop using a legacy DTV tuner?

Trip

If you want a loaner tuner to toy with, let me know. I can have one sent to you at some point in the near-ish future.

(I'm much too tired from moving into school today to comment on the rest of that exhaustive message.)

- Trip

__________________
KJ4IEA

Comments are my own and not that of my employer or anyone else.

Boo-Ray

@Jsmar: What would one do with the M/H data/programming?

jsmar

Potentially you would be able to choose a mobile program you wanted to watch, and either have it streamed to media player like VLC or extracted to a file in a format that could be played by a media player.

Boo-Ray

Understood. I'd be interested in the overall breakdown of ATSC-M/H, but I have no interest in recording and or watching on media player. I think we've gone a bit off subject here from the OP.

t_newt

jsmar, what you are doing is cool--I like the way ATSC-M/H could be used to receive television in weak signal areas, for people who've lost over-the-air due to ATSC conversion.

I just happen to have a WinTV-HVR1800 sitting around. If you are interested I could send it to you. I'd rather it be used by someone doing something cool than just sitting around collecting dust.

Trip

t_newt:

Curiosity strikes; where are you located?

- Trip

__________________
KJ4IEA

Comments are my own and not that of my employer or anyone else.

jsmar

Thanks. Just to be clear, what I am doing won't be useful to that class of people, i.e. if I succeed my software would only facilitate reception of ATSC M/H reception for those who are able to get good reception of normal ATSC programming with their legacy receivers. In order to be able to take advantage of the increased reception ability of ATSC M/H you will need a true ATSC M/H receiver.

Thanks, but I won't actually need a tuner like that unless I succeed in the first phase of decoding the ATSC M/H data.

otaota

Yes, you've got other interested parties listening! Thanks for sharing.

I also try to follow the ATSC discussions and it's great to hear from someone actually doing some real implementation.

Sooner or later we'll see a finalized A/153 standard and chipsets hit the market, so I'm guessing the M/H programming streams will be readily available then. I think the value of the work you're doing is mostly in:

1) independent understanding and clarification of the standard (all the real stuff you need to know about implementation that is not written in the spec). I'm sure there will eventually be the need for many updates and addendums to the standard to fill in these missing details.

2) a diagnostics/monitoring tool. You will be able to provide information about the M/H structures at a much lower level than what is probably exposed by commercial M/H chipsets. It may be possible to put together a poor man's protocol analyzer as an alternative to the rare and expensive equivalents from the likes of Harris or Rhode & Schwarz.



Have you ever looked into the USRP project? There are already several people working on ATSC decoding with that platform and M/H is likely to be tackled soon, if not already.

Cheers,
Chuck