Machine learning augmented reverse engineering Masters thesis

Machine learning augmented reverse engineering Masters thesis

Malicious software is a burning problem in todays ICT industry. There is an ongoing cat-and-mouse game between malware authors, who are writing more sophisticated code every day and returning to life old malware with new protection methods, and security researchers who are struggling to analyze malicious code and find a way how to stop it. Sophisticated malware has multiple defense layers which serve to fool anti-virus (anti-malware) software, malware analysis platforms, as well as to make it harder for security researchers to analyze malware internal workings. Security researcher has to go through mostly manual process of reverse engineering malware, document malware internal workings, write malware signature (or some other form of malware detection technique) and to see if the damage malware has caused is reversible and how to remove it from the system. By the time that is done malware has already caused often irreversible damage (like a stolen bank account).

Malware, besides the malicious code itself, often contains garbage code, which essentially does nothing and its sole purpose is to confuse security researcher, harmless code, which is not important for understanding malware workings and can be safely skipped over during analysis, as well as various anti-debug and anti-disassembly techniques which are meant to make malware impossible to analyze. Security researcher loses precious time while fighting his way through various anti-debug and anti-disassembly techniques and analyzing code which turns out to have nothing to do with malicious activities. The more time researcher spends analyzing garbage code and defenses, the more damage malware does to users and/or companies. It would be useful to have a system which would help guide researcher while analyzing malware, which could tell him not to waste time on some part of a code because it probably is not important.

As a part of my masters thesis I am going to develop a prototype system which could be used to help shorten the time needed for a researcher to analyze malicious code. Machine learning techniques are going to be used to try to predict if the segment of code is relevant to the current analysis. Researcher leaves plenty of data while doing reverse engineering, and it needs to be seen if that data could be used to teach a classifier whether its okay to skip over CALL or JXX instruction or should researcher analyze that part of the code because it contains important functionality. It should also be investigated if that kind of a system would be able to help researcher to defeat malwares protections.

The idea is to build a system which would be able to learn by watching researcher doing reverse engineering of a malware. System learned in that way would then for each CALL and JXX instruction suggest two options to a researcher:

• CALL/JXX instruction jumps to the part of a code which is relevant to the analysis and should be investigated,
• CALL/JXX instruction jumps to the part of a code which is not relevant to the analysis and it can be stepped over.

There are two possibilities if the system is not sure (probability that code segment is either important or it is not important is low) whether the code segment is important for analysis or not. First is that it tells it cannot decide should code segment be analyzed and the second is that it offers no output. The system should learn and improve itself during every reverse engineering session.

The architecture is not yet designed but it is known that the system is going to consist from two components, front-end and back-end.  Front-end is going to be implemented as a plugin to one or more popular debuggers such as ollydbg. Its job description is:

• record researchers action and program state from which system can make classifications and learn,
• notify back-end if the classification is wrong (researcher skips over function which is classified important or he analyzes function which is marked non important),
• display classification result to a researcher.

 During the research phase back-end is going to be separate entity on the same computer, but in the later phase its probably going to be located on a server or part on a server and part on a client computer. Its job description is:

• conduct learning based on data received from front-end,
• conduct classification based on data received from front-end.

System needs to be designed to be able to accept ways of different feature extraction simply and it needs to be able to accept multiple classifiers to see which gives the best generalisation capacity with regards to a given problem. It needs to be decided where the program state processing and feature extraction is going to be located with regards to speed and load. I am going to describe each of components in more details as it is developed.

download file now

download
alternative link download