Greybox Program Synthesis: A New Approach to Attack Dataflow Obfuscation

August 6, 2021

https://www.blackhat.com/us-21/briefings/schedule/index.html#greybox-program-synthesis-a-new-approach-to-attack-dataflow-obfuscation-22930

Quote

Obfuscation is getting broadly adopted for a wide range of applications and especially to protect intellectual property (IP) in mobile ecosystem (Android, iOS) and embedded systems at large. It is now ubiquitous, and everyone is unwillingly and unknowingly executing obfuscated code. Throughout adoption it also gained maturity, potency making assessing such protection incrementally harder.

It is used in a variety of contexts from malware to famous and widely used mobile applications. In either case, the goal is to protect software secrets, communication protocol, APIs, and its inner working from reverse engineering. Thus, finding new ways to defeat evolving obfuscation schemes is getting more and more important in this endless cat and mouse game.

This talk presents the latest advances in program synthesis applied for deobfuscation. It aims at demystifying this analysis technique by showing how it can be put into action on obfuscation. Especially the implementation Qsynthesis released for this talk shows a complete end-to-end workflow to deobfuscate assembly instructions back in optimized (deobfuscated) instructions reassembled back in the binary.

More specifically the talk presents the greybox synthesizer developed combining two core components, an I/O-based black-box synthesis using precomputed tables and a white-box AST search algorithm backed by symbolic execution. This new approach provides a very good trade-off between accuracy and speed. Various experiments to improve it like expression linearization, expression learning or table evaluation JITing will be presented with both their strengths and weaknesses to address obfuscation schemes attacked.

Among existing schemes to impede program understanding, we show results obtained on various transformations like Mixed-Boolean-Arithmetic (MBA), arithmetic encoding, or virtualization that originates from multiple obfuscators like Tigress, YANSOllvm, or commercial applications.

Finally, we will highlight limitations of the approach, open research problems yielded, and various insights on how to improve the algorithm to bypass roadblocks in order to better leverage program synthesis for deobfuscation.