Okay, first of all he never said "I'm retarded" he said "I feel like doing paperwork today", and the wait was closer to 10 seconds, nowhere near the 3 minutes you say.
Yes, he did unassign your job, but from what I can tell he was trolling, and your reaction to jump on the table and try to smash...