This thesis is focused on a study of security issues related to an execution of cryptographic algorithms in an untrusted environment. It mainly studies whitebox cryptography methods of transforming algorithms in such a way they resist attacks like key-extraction and inverting in some extent. Particularly it examines whitebox transformations of AES cipher and attacks on these transformations. Transformations construction and implementation is described. In the thesis was discovered the known attack works also on AES transformation using dual ciphers by Karroumi that was supposed to resist the attack. The new improvements for increasing a resistance of transformations to known attacks were proposed.