Mevcut Yöntemlerin Sentezi ile Web Sayfalarının Belli Yapılardaki Bölgelerinin Tespiti. - Sirat, Malika; Özyavaş, Adem (Electronic Turkish Studies, 2022)
Web content mining is a branch of web mining where the content usually consists of html tag-data and more importantly structured data such as lists and tables with html tag-data and possibly other nested structured data. Usually, useful pieces of information reside in these structured data regions following some pattern. But due to noises that break this pattern, detection and extraction of useful data records turn out to be a challenging task. For data record detection and extraction applications, any non-regular html tag in the structured data region as well as any data outside of the structed data regions of interest are considered to be noise. The proposed work is a synthesis of existing methods combining their successful components with minor modifications to further improve the accuracy performance of the extraction systems. Most of the existing methods utilize the Document Object Model (DOM) tree to cluster similar nodes to detect the data record regions or convert DOM tree into a Tag Path Sequence (TPS) to represent a web page before detection. The proposed work first generates the unique sequence that encodes a webpage and utilizes the Suffix Tree-based Extraction Method (STEM) to extract data record patterns. It then clusters subsequences based on the assumption that data records that are related are similar in structure. The similarity between sequences are computed using the longest common subsequence (LCS) algorithm. In the last step, the noisy subsequences are removed by pruning the corresponding nodes from the DOM tree. Some publicly available web pages and Testbed dataset are used for testing the proposed work. The obtained results are compared with those of the TPS filter method. Experiments show that the average number of noise removed using the proposed work is 43% of the DOM tree, which can match previous reports and it is higher than that of TPS filter.