Fair question--it's mainly due to a couple of things:
1) Initially I thought I was going to end up a bit short when mounting the grips, switchgear, and mirror mounts back on the bars due to the slightly shorter straight length of the handgrip ends of the bars, so I pushed the switchgear in as far as I felt I could and still get the mirror / lever mounts to install properly. As they are now, they're just starting to creep onto the bend in the bar, but seem to work out OK.
2) At the time since I thought I might end up short (which, due to the washers I had to place between the handguard end mounts and the threaded adapters, I did), I shortened up the Oxford grips by about 1/8", which as it turns out I probably didn't have to.
Is the fit of everything on the bars 100% perfect? Maybe not quite, but for my hands the comfort is worth those couple of minor anomalies. If I did it again, I could probably slide the switchgear back outboard a little to minimize the gap you point out, but not the mirror/lever mounts. I thought about making or having someone make a couple of spacers to put between the handguard adapters and the bar ends to space them out, but for now the washers seem to do the trick.
Regarding the grips themselves, if I got a replacement set of Oxford grips (these are the "Premium Adventure" version) I would not bother to trim them--and that would address at least the visual part of the gap you asked about.