I have -38 karma on WF..... It really doesn't matter. Karma is a bullshit add on to proboards sites. And maybe people just found you annoying at times, and decided to smite you. I'm sure it wasn't one person just destroying you over and over again, more like a mixture of people. Karma doesn't matter, forget about it.
-21 was me, i lowered it from -12. However before I lowered it, it was at -12 and none of those were by me, I could tell you who did it but I don't think they'd appreciate me saying so and it'd probably cause arguments.