Hi Georg,
as of now, RServe is not supported to be installed on the same hardware as the productive SAP HANA appliance.
And yes, the data transfer is a bottleneck here and sure enough pumping data over the network isn't helping here either.
The goal obviously needs to be (and is) to get the data processing logic into SAP HANA into the indexserver process.
That's what the AFL/BFL/PAL functions are for.
Current state is that you cannot just go on and write your own functions on that level.
I've no idea on whether or when this restriction can or will be lifted, but hey: I'm not part of the program management for SAP HANA 
...
- Lars