TrueCrypt’s response to the so called ‘janitor attacks’ is pretty straight forward - as long as someone had physical access to your hardware you should assume the worst if truly paranoid. Moreover, according to the developer, the physical security of the hardware is not TrueCrypt’s problem, and that a good strongbox might offer a clue that the hardware has been tempered with in the absence of its owner.

As with so much of Security, "Don't assume" is a great starting place ...