Title: Effective Deep Learning Primitives Design for Binary Vulnerability and Malware Detection

As deep learning continues to achieve success in various areas, it has also attracted significant attention in cybersecurity. While deep learning is highly effective at detecting cybersecurity threats, existing approaches often depend on existing deep learning primitives. Here, “deep learning primitives” refer to functor x-objects designed to encapsulate particular computations, including data pre-processing, model architectures, and data augmentation. Although the design of deep learning primitives has the potential to enhance performance in binary vulnerability and malware detection, this area remains largely unexplored. In this dissertation, we introduce effective deep learning primitives optimized for the cybersecurity domain to address two critical areas, which are smart contract vulnerabilities and malware detection. As an emerging technology in decentralized finance, smart contracts need robust vulnerability detection, while malware remains an evolving and resilient threat. By considering both as binary analysis tasks, deep learning is possible to identify smart contract vulnerabilities and malware in traditional binaries effectively. Specifically, we obtained three remarkable contributions. First, a code-targeted architecture and data pre-processing strategy for binary vulnerability and malware detection. Second, a de-obfuscation technique designed to address obfuscation techniques that commonly degrade malware detection performance. Third, a data augmentation method tailored for smart contracts, which is a relatively new programming language that currently lacks extensive datasets. The primary contributions of this dissertation are as follows:

Deep Learning Architecture for Effective Binary Vulnerability and Malware Detection : Deep learning models, particularly those utilizing convolutional neural networks (CNNs), have demonstrated high detection accuracy and speed in binary code classification. However, codes, which are converted into image-like representations, often ignore the meaning and context of binary codes, resulting in false detections. To address this problem, we introduce CodeNet, an efficient code-targeted CNN architecture designed to detect smart contract vulnerabilities and malware while preserving code semantics. Additionally, to improve learning efficiency, we propose a data pre-processing method to convert smart contract code and malware into efficient features. Experimental results indicate that the proposed CodeNet architecture offers both high detection accuracy and competitive processing times.

De-obfuscation as a Data Pre-Processing for Binary Malware Detection : One of the simple approaches to detect malware using a deep learning model is to convert it into binary form. However, an obfuscation technique, which is commonly used to hide malware, significantly alters critical malware features, leading to misclassification in the detection model. To solve this problem, we propose a generic API de-obfuscation and unpacking method called GUARD. GUARD combines emulation-based obfuscated call detection with an analysis algorithm and a scattered import address table (sIAT) to restore original API calls from packed files effectively. Evaluations against advanced commercial packers, including Themida and VMProtect, demonstrate that GUARD can restore obfuscated APIs and unpack files, achieving up to a 24\% improvement in malware detection rates.

Data Augmentation to Improve Smart Contract Vulnerability Detection : The programming of smart contracts inherits numerous software development challenges, including security, reliability, and data efficiency. Recently, deep learning-based solutions have been in the spotlight in software development to solve various challenges. A deep learning model recognizes patterns from data, and it is widely known that the number of training sets improves accuracy and results in better. However, it is challenging to obtain a public dataset that is perfectly suitable for a specific task. Compared to the large amount of datasets available for conventional programming languages like C/C++, JAVA, and Python, the data size for smart contracts is relatively small. This scarcity is particularly challenging as smart contracts are a recent innovation compared to traditional programming languages. To address this problem, we propose the Compiler-Guided Generation Network (CGGNet) to augment smart contract datasets. Unlike traditional text generation methods like SeqGAN, CGGNet employs a compiler as an oracle within generative networks to guarantee the validity of the generated smart contracts. Incorporating Monte Carlo tree search, CGGNet improves the diversity and validity of generated contracts, effectively surpassing GAN-based models in generating syntactically correct augmentations. This approach enables the generation of millions of unique and valid smart contracts from thousands of inputs, effectively addressing underfitting in deep learning applications.

Three deep learning primitives described above in this dissertation provide improvements in analyzing binary smart contracts and malware. Therefore, we can use these primitives to enhance binary vulnerability and malware detection accuracy.