“Dirclustering”: a semantic clustering approach to optimize website structure discovery during penetration testing

Dirbusting is a technique used to brute force directories and file names on web servers while monitoring HTTP responses in order to enumerate server contents. Such a technique uses lists of common words to discover the hidden structure of the target website. Dirbusting typically relies on response codes as discovery conditions to find new pages. It is widely used in web application penetration testing, an activity that allows companies to detect website vulnerabilities. Dirbusting techniques are both time and resource-consuming, and innovative approaches have never been explored in this field. We hence propose “Dirclustering”, an advanced technique to optimize the dirbusting process by leveraging semantic clustering. Specifically, we use semantic clustering techniques to organize wordlist items in different groups according to their semantic meaning. The created clusters are used in an ad-hoc implemented next-word intelligent strategy. This paper demonstrates that clustering techniques outperform the commonly used brute-force methods. Performance is evaluated by testing eight different web applications. Results show a performance increase that is up to 50% for each of the conducted experiments.