In order to obtain a standard content, you must work hard on collection! Write the collection rules.
My measure of standard content:
Each paragraph is <p></p>
No redundant HTML tags and irrelevant characters
How to extract data
Select regular extraction and fill in the combined result <p>[Parameter 1]</p>
Some content does not have p tags at the beginning and end. Line breaks are done with br tags, so we need to create p tags.
Data processing
Step 1: Remove content irrelevant to the text, such as advertisements, comments, etc.
Regular <header>[\s\S]*?</header>|<!--.*?-->| |Rule 4|Rule 5 is replaced by empty
If published to some free platforms, some HTML special characters will not be converted. Just add '|&.*?;' at the end to filter out the last special HTML characters.
Step 2: Keep only images and commonly used block-level elements
Regular (?i)<(?!/?h|/?p|/?div|br|img).*?> is replaced by empty
Only retain the title tag h segment tag p, div, br and image tag img; the preceding (?i) indicates that it is not case sensitive
Step 3: Remove the selectors or styles of all tags except the img tag, and change the tag to p
Regular <(?!img)(/?)\w+.*?> is replaced by <$1p>
Step 4: Change the messy image styles into standard image codes
Regularly replace <img.*?src="(.+?)".*?> with <img src="$1">
Step 5: Standardize paragraph tags <p>Start</p>End
Regular </?p> replaced with </p><p>
Paragraphs on HC.com only have a <p>start and no</p>end
For some articles on some sites, there is no <p> after the end of </p> and the content of the next paragraph begins directly.
Step 6: Replace the spaces before and after <p> or </p> with empty spaces
Regular \s*(<\/?p>)\s* is replaced by $1
Step 7: Replace more than 2 consecutive <p> or </p> with 1
Regular (<p>){2,}|(</p>){2,} is replaced by $1$2
Step 8: Replace all empty paragraphs with empty paragraphs
Content <p></p> replaced with empty
Step 9: Due to step 5, the beginning of the article may be </p> and the end may be <p> and they need to be cleaned up
Regular ^</p>|<p>$ is replaced by empty
Filter English regular expressions [a-zA-Z]
Regular filtering of numbers with more than two digits (\d{2,100})
