My first thought is that the shunt is being bypassed when using the portable charger. I'm not an engineer or an electrician, so decided to wait until someone more knowledgable responded. I agree with Pogen's advice. I have done a lot of boat system wiring and it is always helpful, if not essential to draw a schematic. Then go read the Victron installation instructions.
Tom P.