I managed to get a hold of a PS/2 keyboard + USB power supply and started testing, and it seems that it's my old breadbox that fucks up things..
I disasembled my non-working C64 (newer model) and fixed it today, plugged in the Chameleon and tadaaa... works..
So, I am a happy camper at the moment.. but still puzzled on what is wrong with my breadbox, as it works flawless without the Chameleon.. ?