Practical Guide to Handling Prohibited Keywords in Website Advertising: Python-based Scanning + Baota Firewall Filtering + Keyword Database Management
authorHong Mu (QQ: 1810216796)
first publishHongmu Notes
GitHub：https://github.com/1810216796(Complete source code)
background
Recently, we received feedback from a client stating that their website had been summoned for regulatory consultation and fined by the market supervision authorities due to the inclusion of prohibited advertising terms—such as "First," "Best," and "National-Level" —on certain pages. These terms were scattered across hundreds of pages; manually identifying them would be time-consuming and labor-intensive, and there was a high risk of oversight. To address this challenge, we developed an automated scanning system that uses Python web crawlers to perform recursive scans across the entire website. Combined with a sophisticated list of prohibited terms and a whitelist mechanism, this system conducts daily automated full-site scans and generates a visual monitoring dashboard, ensuring that any non-compliant content is fully exposed.
This article provides a comprehensive overview of the entire solution—covering everything from the underlying principles to deployment—along with all the associated code and practical usage insights.
Easy Overall Approach
- Vocabulary ConstructionThis tool helps identify and remove absolute terms, medical efficacy claims, financial commitment statements, and other prohibited expressions as explicitly banned by the Advertising Law; it also incorporates a contextual whitelist to prevent misclassification (for example, the phrase "World-Class Cultural Heritage" is considered an objective description and does not constitute a violation).
- Web crawler scanA Python script that uses multithreading to concurrently scrape all pages of a website, extract visible text, identify prohibited terms, and record the violating pages along with their surrounding context.
- Incremental & Full Data LoadDefault incremental scanning (skipping URLs already scanned), triggered once every day at midnight;
--resetFull scan – ensures all new pages are covered. - Monitoring GuardianThe Shell script checks the scanning process once per minute; if the process becomes stuck, it will be automatically restarted, ensuring stable operation 24/7.
- Visual PanelThe PHP page displays in real time the number of scans performed today, the count of violations, the system status, site-specific statistics, and a aggregation of prohibited terms, enabling operations and maintenance personnel to monitor these metrics at any time.
- BaoTa Firewall IntegrationThrough;
rules.txtLexicon: Automatically updates BaoTa Firewall configurations to replace prohibited terms in real time during access, enabling filtering at the source.
Environmental Preparation
- serverCentOS / Ubuntu; 2 CPU cores and 4 GB or more RAM (recommended minimum RAM: ≥ 2 GB)
- Python 3.6+Required dependencies:
pip3 install requests beautifulsoup4 lxml
- PHP 7.4+(For monitoring panel)
- Nginx / Apache(Managed PHP Panel)
- Pagoda Panel(If you need to use the Firewall Replacement feature)
Directory Structure (GitHub Repository)
ad-violation-scanner/├── sacn_site.py # 核心扫描脚本（支持增量/全量）├── index.php # 实时监控面板├── update_ads.py # 宝塔防火墙词库更新脚本├── rules.txt # 替换词库（违禁词=替换词）├── 宝塔定时任务.sh # 一键创建定时任务├── scan_monitor.sh # 进程守护脚本（可选）├── .user.ini # PHP open_basedir 限制└── README.md # 详细使用说明
The source code is available as open source on GitHub:https://github.com/1810216796
Python Scanning Script (sacn_site.py)
Core Principle
- make use of
ThreadPoolExecutorControl the number of concurrent connections (default: 20) to balance request rate and server load. requests.SessionMounting retry mechanism: 45-second timeout with 2 automatic retries.- Extract visible text from each page (excluding <script> and <style> tags) and identify prohibited words using regular expression matching.
- Context-based whitelist filtering helps avoid false positives such as "Peak Period" or "Top Rank Nationwide (Geographic Description)".
- recursively crawl all pages under the same domain, filtering out static resources (e.g., CSS, JS, images, etc.) and specified parameters.
- Save scanned URLs in real time (
scanned_urls.txt) and violation records (violations_results.txt), supports interrupt recovery. - Generated upon completion of scanning;
index.htmlSummary Report.
key parameter
| parameters | explain |
|---|---|
max_workers | Number of concurrent threads (default: 20; adjustable) |
session.timeout | Request timeout duration (seconds) – set to 45 |
Retry(total=2) | Number of retries – to mitigate network jitter |
--reset | Clear history – Full scan |
Prohibited Word List and White List
- Prohibited Word ListOver 200 keywords, including absolute terms in advertising laws, medical claims, financial promises, and prohibited terms in the tourism industry.
- white listUsing regular expressions to match contexts—such as "World-Class Cultural Heritage" or "The Nation's Highest Peak" —does not constitute a violation.
Process Monitoring and Scheduled Tasks
scan_monitor.sh (Optional)
Check the scanning process once per minute:
- If the process does not exist, incremental scanning will be automatically initiated.
- If more than 24 hours have passed since the last full scan, the task will be executed automatically;
--resetFull scan. - If the system detects a deadlock (log timeout without update or no increase in the number of pages), it will automatically restart.
CRON tab configuration
# 每分钟监控扫描进程（需要 scan_monitor.sh）
* * * * * /bin/bash /path/to/scan_monitor.sh >> /path/to/monitor_cron.log 2>&1
# 每日凌晨2点更新宝塔防火墙词库（替换规则）
0 2 * * * /usr/bin/python3 /path/to/update_ads.py >> /path/to/update_cron.log 2>&1 PHP Real-Time Monitoring Panel (index.php)
function
- real-time dataToday's scanned page count (Read;
scanned_urls.txt(Number of rows), Number of violations today (Statistics);violations_results.txtChina;URL:Count). - running stateBased on;
scan.logThe last modification time is used to determine whether the process is still alive. - Site OverviewDisplay the number of pages scanned at each site, the number of prohibited word types, the number of problematic pages, etc.
- Prohibited Word AggregationClick on a website to view all prohibited terms and their occurrence frequency; clicking on a term allows you to view the detailed context.
- Automatic RefreshData is updated via AJAX every second – no manual refreshing required.
Deployment
support index.php Place it within the website's directory; ensure that the PHP script is readable and writable within the same directory. .txt ; .log Document. Access http://your-domain/index.php Just go ahead.
Bota Firewall Vocabulary Auto-Replacement (Core Feature)
operational principle
BaoTa Firewall; body_character_string The field supports keyword replacement on response content. Leveraging this feature, we map prohibited terms to compliant terms, which is applied when users access the content.Intuitive ReplacementDoes not affect normal browsing.
Lexicon file: rules.txt
One replacement rule per line, formatted as ; 违禁词=替换词.for instance:
最佳=较佳第一=首先国家级=知名100%=全部治疗=调理
Update the script: update_ads.py
- back-upAutomatically back up the original configuration file to ;
/www/server/btwaf/backup/。 - mergeRead;
rules.txtUpdate;body_character_stringField (uses existing rules). - heavy loadAutomatic execution;
nginx -s reloador ;systemctl reload httpdApply the configuration.
Scheduled execution
It is recommended to run this process once every morning to keep your word list up to date.
0 2 * * * /usr/bin/python3 /path/to/update_ads.py >> /path/to/update_cron.log 2>&1 Practical Deployment (Full Steps)
1. Cloned Code
git clone https://github.com/1810216796/ad-violation-scanner.git
cd ad-violation-scanner2. Install Python dependencies
pip3 install requests beautifulsoup4 lxml3. Configure scanning target
edit sacn_site.pyEdit; WEBSITES List:
WEBSITES = [
{"name": "客户网站", "urls": ["https://www.example.com/"]},
]4. Configure Synonym Database
edit rules.txtAdd the prohibited words that need to be replaced along with their corresponding replacement words.
5. Configure a scheduled task
chmod +x 宝塔定时任务.sh
./宝塔定时任务.sh
# 按提示选择 1（监控守护）或 2（防火墙更新）6. Configure Web Access
support index.php ; .user.ini Place it in the website's root directory and ensure that the directory permissions are correct.
7. First-time manual startup (optional)
nohup python3 sacn_site.py > scan.log 2>&1 &The monitoring script will take over in the next minute.
Effect Display
- Monitoring PanelReal-time display of the "Today's Scans" and "Today's Violations" counters, updated every second.
- Non-compliant aggregationClick on a site to see all prohibited terms at a glance; click on a term to directly view the corresponding page and context, making it easy to locate and modify content.
- Scan ReportAutomatically generated upon completion of each scan;
index.htmlContains site information, URLs, and prohibited word details; can be archived for future reference. - Firewall ReplacementWhen accessing a page, prohibited terms are automatically replaced without any user notice, significantly enhancing compliance.
⚠matters need attention
- concurrency controlAdjust server performance;
max_workersAvoid setting the value too high, which could exhaust the target website or local resources. It is recommended to start at 10 and gradually increase it. - Lexicon MaintenanceThe Advertising Law is subject to continuous updates and requires regular maintenance;
rules.txt;EXACT_FORBIDDEN_WORDSPrevent missed or incorrect rulings. - Timeout and RetryingWhen the network environment is unstable, you may appropriately increase the value;
timeout;totalNumber of retries; however, avoid infinite retries. - BaoTa Version：
update_ads.pyDeveloped based on Baota 7.x; field names may differ in other versions – please back up and verify the system before use. - Access Control SecurityIt is recommended to set an access password or restrict access to the internal network only on the monitoring panel to prevent the leakage of sensitive information.
frequently asked questions
Q: What should I do if the scanner doesn't move?
A: The monitoring script will restart automatically (if configured ); scan_monitor.sh), or can be done manually; pkill -f "python3 sacn_site.py" Terminate process.
Q: Does replacing the firewall not take effect?
A: Inspection; rules.txt Is the format correct? (= (No leading or trailing extra spaces); confirm that the Torre Firewall has the "Response Content Replacement" feature enabled, then reload the Web service.
Q: Does the PHP dashboard display "Today's Scan: 0"?
A: Ensure; scanned_urls.txt The file must exist and contain content; check the directory permissions (PHP must have read access), and confirm that the scanning script has been executed.
Q: How to perform an incremental scan without resetting?
A: Execute directly; python3 sacn_site.py(Without; --reset) enables incremental scanning, automatically skipping already scanned URLs.
epilogue
This solution has been successfully deployed across numerous client projects, having scanned hundreds of thousands of pages, and has effectively helped mitigate advertising compliance risks. The core principle isPrecise vocabulary database + Context-based whitelist + Automated monitoring + Firewall integrationThis not only reduces labor costs but also enhances compliance.
If you have similar requirements, feel free to refer to this article for implementation. The source code is available as open source on GitHub – feel free to Star it, Fork it, or raise an Issue.
authorHong Mu
QQ：1810216796
GitHub：https://github.com/1810216796
This article was originally published on Hongmu Notes; please credit the source when reposting.
