Why are Agile approaches not proving effective in completing R&D projects on time? Is it a nightmare for those Project Managers who are leading such ventures?

First let me say that the Agile Practice Guide is not a good source. It was created with the Agile Alliance and the Agile Alliance is focused on software so the guide could help to software projects. Second, Agile is not about to use a method or process, did not start with the Mainifesto,Agile was born inside the USA DoD NSF/Agility Forum in 1990 but their practices started time before.Third, Agile applies to R&D initiatives (I used it and when you search for Agile history you will see that) but as into any other type of initiatives there is something mostly missing: the first thing to do is an impact analysis taking into account the whole enterprise architecture to understand if the organization is ready to use Agile. Agile fits for R&D initiatives mainly when there is some type of uncertainty about the product and the acceptance into the market and some aditional information is needed. Sometimes after the first version, depending on the product characteristics,next versions are using other methods. Remember: just in case organization need to use Agile based method the Agile world does not end in Scrum. You have other methods like DSDM that are best suited for R&D initiatives (just after you see it when impact analysis ends).