I've had my R5 and 100-500 for a few weeks now (coming from Nikon D500 and 200-500 lens). I have very limited experience with the 1.4x extender so take this with a grain of salt. I rented the 1.4x extender and the RF 100-500 for 2 days before buying my R5. In that limited time, as long as I had enough time to find the subject and for the camera to focus, I didn't notice any significant slowing of AF or loss of IQ when I had the extender on vs. when I didn't. For me, it was just getting used to quickly getting the flying bird into frame given to the 700mm reach....I just need more practice with that. I've not tried the 2x extender.
The RF 100-500 is a very good lens. I haven't tried the 600 or 800 yet. I think it ultimately depends on how often you want/need to be able to zoom in/out and how often you're in good light. In case you don't already know, because of the lens design, you have to extend the 100-500 beyond 300mm to be able to attach an extender. So with a 1.4 extender, the lens becomes a 420-700mm zoom f/8-10 lens (or with the 2x extender, a 600-1000mm f/11-14 lens).
Also, consider your style of shooting. I tend to walk/hike and shoot what I see, so I usually like a lighter system. The 600 and 800mm is lighter, but the 100-500 isn't too bad either for weight. I carried my R5 and 100-500 for almost 4 hours on Monday slung over my shoulder while walking through hilly terrain on packed snow/ice and the weight didn't bother me. If you're concerned about weight, then the 600 or 800 mm may be the better option assuming you shoot in locations with good light and don't find yourself needing to zoom. I will say, though, that these cameras have quite good high iso capability and you can recover quite a lot in post.
I've gone back and forth about this and still am not sure. I ended up purchasing both. The 1.4 extender will be here this week. My camera store said the 800mm lens will be on backorder for a while so I may end up cancelling the 800mm if I like the extender enough. There are many locations that I go to that I need the ability to zoom so I may just stick with the 1.4x extender on the 100-500.
Not sure I helped or just gave you a more circular argument, but misery loves company and I'm right there with you!