I mainly only use DVD decryptor, FYI it hit end of life at 3.5.4.0 when they got sued (Sony I think) and shut down. I also used DVD shrink 3.1 (old) just fine to fit to a single layer disk when I didn't care about quality, it's fast and easy. Otherwise I am using the big three method (DoItFast/CinemaCraft/Senarist) to get them down to 4.3gig. DVD decryptor is going to be the superior program for getting that data off the disk and onto an image. I would recommend Nero 7 for dual layer burning at this time but check out their site to see when they implemented that automatic positioning of the layer break feature. I'm not on dual layer yet, probably have 80 DVD-R's still.
Because you are playing back on a set top player I will relay my experiences with a Panasonic DVD-RP56 (secretly had the Sage/Faroudja chipset for a really good deinterlace). This player was very picky with the media I fed it with, and media that it was rather unhappy about reading would sometimes be OK if burnt at a slower speed like 4x instead of 8x or 12x. The failure mode was it would play back jerky (I guess that's dropped frames?) and would progressively get worse / stop playing entirely toward the end of the disk. This sounds like what you are seeing? I think it got worse because of the way the disk was burnt with the burners stepping up their speed throught the disk. At any rate even if you think you have success, skip to the credits at the end of the movie and if that plays fine you are good. This looks to be where you are now, with a picky player.
Unfortunately I think you are going to have to try to drop your burn speed if you can and try different media. I have no experience with DL media but for single layer I found my player liked Yaiyo Yuden (SP?) the best, was able to burn that at 8x. In your position I was not truly happy until I replaced my set top player.
My quick review of the DVD players I have owned:
Panasonic DVD-RP56 - Excellent picture thanks to Sage/Faroudja chipset, component output very good. Mine had difficulty with burnt media.
Phillips DVP642 - Plays Divx and lots of other standards. Reads ANY disk you throw at it, even stuff I dug out of the wastebasket from the RP56 days. Component quality not suitable for a screen over 30" in my opinion. The remote blows in ergonomics (no eject to boot) and was not learn-able by my universal remote so I was stuck with it.
Zenith DVB318 - DVI or component out, with firmware updates available for even better quality. Has the new Sage/Faroudja chipset (now totally sold on these guys). This is an upconverting player, and while my TV does it's own upconversion if necessary the upconversion on this at 1080i is superior. Been out for a while, should be easy to find.
A couple links for further info:
http://www.avsforum.com/
http://www.doom9.org/ <-- Will help a lot with the backup process and the programs you are using.
Jeezum crow that was a lot of typing, peace out.