In web projects, page caching is a commonly used technique for improving performance. Previously, I was using a simple file caching plugin [Empire cms PHP caches web page code to speed up access】It caches all pages equally. However, in real-world scenarios, some pages (such as the home page) receive a high volume of daily traffic, while other pages (such as less popular articles) may only be accessed a few times per day.
This "one-size-fits-all" caching strategy has led to the accumulation of a large number of low-value cached files, consuming disk space and inode resources. I need a solution.Intelligent Cache Cleanup MechanismDecides whether to retain or delete pages based on their access frequency, while ensuring that the total number of cached pages does not exceed a predefined upper limit.
II. Overall Design Approach
2.1 Cache Storage
Use a file cache by leveraging PHP's capabilities;
ob_startCapture output and store after compression.Files are stored separately based on their URL's MD5 hash value, preventing a single directory from containing an excessive number of files.
directory structure:
cacheFile/{md5最后1位}/{md5最后2位}/{md5}.phpA total of 16 + 256 directories.
2.2 Access Count
Whenever a cache hit occurs, record the number of page accesses in MySQL.
The counting period is 7 days; caches with a duration exceeding 7 days but a number of accesses below a threshold (e.g., 20 accesses) will be cleared.
2.3 Cleanup Strategy (Three-Phase Approach)
Phase 1Delete caches that have exceeded the retention period but have insufficient access counts (directly interacts with the database and files).
Phase IIIf the total number of cached items exceeds the upper limit (e.g., 1,000,000), delete the items in order from oldest to newest until the target count is reached.
Phase 3Scan the file system to resolve inconsistencies between the database and the file system (by adding missing records or deleting redundant records), and control the scanning speed to avoid impacting the server.
III. Database Design
Just one table with very few fields:
CREATE TABLE cache_stat (
url_md5 CHAR(32) NOT NULL COMMENT 'MD5值',
access_count INT UNSIGNED NOT NULL DEFAULT 0 COMMENT '访问次数',
cache_time INT UNSIGNED NOT NULL COMMENT '文件修改时间戳',
PRIMARY KEY (url_md5),
INDEX idx_cache_time (cache_time),
INDEX idx_access_count (access_count)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;cache_timeStore files;mtimeUsed to determine whether a timeout has occurred.Queries for Phase 1 and Phase 2 of the Index Acceleration Phase.
IV. Core Code Implementation
4.1 Cache the main file; cache.php
This document is located at the project entry point (e.g., Empire CMS); e/ Table of Contents); responsible for intercepting output, generating caching, and updating counters.
<?php
/******************
* 缓存主逻辑(新目录结构)
* 所有文件统一放在 cacheFile/ 下,按 md5 后 1/2 位分两层
******************/
if (1 == 1) {
// ========== 可配置参数 ==========
define('CACHE_ROOT', dirname(dirname(__FILE__)) . DIRECTORY_SEPARATOR . 'cacheFile');
define('CACHE_FIX', '.php');
define('CACHE_COMPRESS_LEVEL', 6);
// 数据库配置(请按实际情况修改)
define('DB_HOST', 'localhost');
define('DB_NAME', 'cache_music');
define('DB_USER', 'cache_music');
define('DB_PASS', 'eb7rydwKyaxRW7Gs');
date_default_timezone_set("Asia/Shanghai");
// 创建缓存根目录
if (!file_exists(CACHE_ROOT)) {
mkdir(CACHE_ROOT, 0755, true);
chmod(CACHE_ROOT, 0755);
}
// ========== 数据库操作函数 ==========
/**
* 获取 PDO 连接(单例)
*/
function getDb() {
static $pdo = null;
if ($pdo === null) {
try {
$pdo = new PDO(
"mysql:host=" . DB_HOST . ";dbname=" . DB_NAME . ";charset=utf8mb4",
DB_USER,
DB_PASS
);
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
} catch (PDOException $e) {
error_log("DB connect failed: " . $e->getMessage());
return null;
}
}
return $pdo;
}
/**
* 增加访问计数(命中缓存时调用)
* 增加 $mtime 参数,确保记录缺失时能存入正确的文件修改时间
*/
function incrementAccessCount($md5, $mtime) {
$pdo = getDb();
if (!$pdo) return;
$sql = "INSERT INTO cache_stat (url_md5, access_count, cache_time)
VALUES (:md5, 1, :mtime)
ON DUPLICATE KEY UPDATE access_count = access_count + 1";
try {
$stmt = $pdo->prepare($sql);
$stmt->execute([':md5' => $md5, ':mtime' => $mtime]);
} catch (PDOException $e) {
error_log("Increment failed: " . $e->getMessage());
}
}
/**
* 插入新缓存记录(生成缓存时),包含文件修改时间
*/
function insertCacheRecord($md5, $mtime) {
$pdo = getDb();
if (!$pdo) return;
$sql = "INSERT INTO cache_stat (url_md5, access_count, cache_time)
VALUES (:md5, 0, :mtime)
ON DUPLICATE KEY UPDATE cache_time = :mtime, access_count = 0";
try {
$stmt = $pdo->prepare($sql);
$stmt->execute([':md5' => $md5, ':mtime' => $mtime]);
} catch (PDOException $e) {
error_log("Insert record failed: " . $e->getMessage());
}
}
// ========== 缓存路径(新结构) ==========
/**
* 获取缓存文件路径
* 目录规则:cacheFile/{md5最后1位}/{md5最后2位}/{md5}.php
*/
function getCacheFilePaths() {
static $paths = null;
if ($paths !== null) return $paths;
$cacheKey = md5($_SERVER['HTTP_HOST'] . $_SERVER['REQUEST_URI']);
$last1 = substr($cacheKey, -1); // 第一层目录(16个可能值)
$last2 = substr($cacheKey, -2); // 第二层目录(16*16=256个可能值)
$cacheDir = CACHE_ROOT . DIRECTORY_SEPARATOR . $last1 . DIRECTORY_SEPARATOR . $last2;
$mainFile = $cacheDir . DIRECTORY_SEPARATOR . $cacheKey . CACHE_FIX;
$paths = [
'dir' => $cacheDir,
'main' => $mainFile,
'cacheKey' => $cacheKey,
];
return $paths;
}
/**
* 生成伪发布时间(近三天内,同页面同一天不变)
*/
function getTime181Value() {
$today = date('Y-m-d');
$url = $_SERVER['REQUEST_URI'];
$seed = $url . $today;
$hash = md5($seed);
$hex = substr($hash, 0, 8);
$offset = hexdec($hex) % 172800; // 0~2天的秒数
$base = strtotime($today . ' 00:00:00');
$fakeTime = $base - $offset;
return date('Y-m-d H:i:s', $fakeTime);
}
/**
* 读取主缓存内容(解压)
*/
function readMainCache($mainFile) {
if (!file_exists($mainFile)) {
return false;
}
$compressed = file_get_contents($mainFile);
if ($compressed === false) {
return false;
}
$decompressed = @gzuncompress($compressed);
return ($decompressed !== false) ? $decompressed : false;
}
/**
* 原子写入缓存文件(先写临时文件,再 rename)
*/
function atomicWriteCache($file, $data) {
$dir = dirname($file);
if (!file_exists($dir)) {
if (!mkdir($dir, 0755, true)) {
error_log("Cache mkdir failed: $dir");
return false;
}
chmod($dir, 0755);
}
$tmpFile = $file . '.tmp.' . uniqid();
if (file_put_contents($tmpFile, $data, LOCK_EX) === false) {
return false;
}
if (!rename($tmpFile, $file)) {
unlink($tmpFile);
return false;
}
chmod($file, 0644);
return true;
}
// ========== 缓存命中处理 ==========
$paths = getCacheFilePaths();
$mainFile = $paths['main'];
$cacheKey = $paths['cacheKey'];
if (file_exists($mainFile)) {
$content = readMainCache($mainFile);
if ($content !== false) {
// 增加访问计数,传入文件修改时间
$mtime = filemtime($mainFile);
if ($mtime !== false) {
incrementAccessCount($cacheKey, $mtime);
} else {
// 获取失败则用当前时间(降级)
incrementAccessCount($cacheKey, time());
}
// 输出替换占位符
$output = str_replace("{time181}", getTime181Value(), $content);
echo $output;
exit;
}
// 损坏文件删除
@unlink($mainFile);
}
// ========== 缓存未命中,生成新缓存 ==========
if (!file_exists($paths['dir'])) {
mkdir($paths['dir'], 0755, true);
chmod($paths['dir'], 0755);
}
function AutoCache($contents) {
global $paths;
if (http_response_code() === 200 && !empty($contents)) {
$compressed = gzcompress($contents, CACHE_COMPRESS_LEVEL);
if ($compressed !== false) {
if (atomicWriteCache($paths['main'], $compressed)) {
// 获取刚写入文件的修改时间
$mtime = filemtime($paths['main']);
if ($mtime !== false) {
insertCacheRecord($paths['cacheKey'], $mtime);
}
} else {
error_log("Cache write failed for " . $paths['main']);
}
} else {
error_log("Cache gzcompress failed for " . $paths['main']);
}
}
return str_replace("{time181}", getTime181Value(), $contents);
}
ob_start('AutoCache');
}
?>4.2 Clean up the script; cache_cleanup.php
Independent CLI script; recommended to be scheduled to run every day at midnight.
#!/usr/bin/env php
<?php
/**
* 缓存清理脚本(高性能版本)
* 阶段1:删除超时且访问不足的记录及文件(每天执行)
* 阶段2:若总记录数超限,删除最旧的记录及文件(每天执行)
* 阶段3:扫描文件系统修复不一致(仅在周四执行,或手动强制)
* - 使用清单文件 + 批量 SQL,性能大幅提升
*/
// ========== 配置 ==========
define('CACHE_ROOT', dirname(__FILE__) . DIRECTORY_SEPARATOR . 'cacheFile');
define('CACHE_FIX', '.php');
define('DB_HOST', 'localhost');
define('DB_NAME', 'cache_music');
define('DB_USER', 'cache_music');
define('DB_PASS', 'eb7rydwKyaxRW7Gs');
define('RETAIN_DAYS', 7);
define('MIN_ACCESS', 20);
define('MAX_FILE_COUNT', 1000000);
// 阶段3控制参数
define('BATCH_SIZE', 1000); // 批量 SQL 操作的行数
define('DB_FETCH_SIZE', 5000); // 每次从数据库获取的记录数(游标)
define('FLUSH_SIZE', 10000); // 清单文件累积写入阈值
define('SLEEP_USEC', 500000); // 每批后暂停 0.5 秒
// ========== 数据库函数 ==========
function getDb() {
static $pdo = null;
if ($pdo === null) {
try {
$pdo = new PDO(
"mysql:host=" . DB_HOST . ";dbname=" . DB_NAME . ";charset=utf8mb4",
DB_USER,
DB_PASS
);
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
} catch (PDOException $e) {
echo "DB connect failed: " . $e->getMessage() . "\n";
exit(1);
}
}
return $pdo;
}
// ========== 文件路径与操作 ==========
function getCacheFilePath($md5) {
$last1 = substr($md5, -1);
$last2 = substr($md5, -2);
return CACHE_ROOT . DIRECTORY_SEPARATOR . $last1 . DIRECTORY_SEPARATOR . $last2 . DIRECTORY_SEPARATOR . $md5 . CACHE_FIX;
}
function deleteFile($path) {
return file_exists($path) && @unlink($path);
}
function getPhpFiles($dir) {
$files = [];
if (!is_dir($dir)) return $files;
foreach (scandir($dir) as $item) {
if ($item === '.' || $item === '..') continue;
$path = $dir . DIRECTORY_SEPARATOR . $item;
if (is_file($path) && substr($item, -4) === CACHE_FIX) {
$files[] = $path;
}
}
return $files;
}
function cleanEmptyDirs($dir) {
if (!is_dir($dir)) return;
foreach (scandir($dir) as $item) {
if ($item === '.' || $item === '..') continue;
$path = $dir . DIRECTORY_SEPARATOR . $item;
if (is_dir($path)) cleanEmptyDirs($path);
}
if ($dir !== CACHE_ROOT && count(scandir($dir)) == 2) {
@rmdir($dir);
}
}
// ========== 数据库记录操作(批量) ==========
function deleteRecord($md5) {
$pdo = getDb();
$stmt = $pdo->prepare("DELETE FROM cache_stat WHERE url_md5 = ?");
return $stmt->execute([$md5]);
}
function insertRecord($md5, $mtime) {
$pdo = getDb();
$stmt = $pdo->prepare("INSERT IGNORE INTO cache_stat (url_md5, access_count, cache_time) VALUES (?, 0, ?)");
return $stmt->execute([$md5, $mtime]);
}
function countRecords() {
return (int)getDb()->query("SELECT COUNT(*) FROM cache_stat")->fetchColumn();
}
function batchDeleteRecords($md5List) {
if (empty($md5List)) return;
$pdo = getDb();
$placeholders = implode(',', array_fill(0, count($md5List), '?'));
$sql = "DELETE FROM cache_stat WHERE url_md5 IN ($placeholders)";
$stmt = $pdo->prepare($sql);
$stmt->execute($md5List);
}
function batchInsertRecords($batch) {
if (empty($batch)) return;
$pdo = getDb();
$values = [];
$params = [];
foreach ($batch as $item) {
$values[] = "(?, 0, ?)";
$params[] = $item['md5'];
$params[] = $item['mtime'];
}
$sql = "INSERT IGNORE INTO cache_stat (url_md5, access_count, cache_time) VALUES " . implode(',', $values);
$stmt = $pdo->prepare($sql);
$stmt->execute($params);
}
// ========== 阶段1 ==========
function phase1() {
$pdo = getDb();
$threshold = time() - RETAIN_DAYS * 86400;
$sql = "SELECT url_md5 FROM cache_stat WHERE cache_time < ? AND access_count < ?";
$stmt = $pdo->prepare($sql);
$stmt->execute([$threshold, MIN_ACCESS]);
$md5List = $stmt->fetchAll(PDO::FETCH_COLUMN);
$count = 0;
foreach ($md5List as $md5) {
$file = getCacheFilePath($md5);
deleteFile($file);
deleteRecord($md5);
$count++;
}
echo "阶段1删除记录与文件数: $count\n";
return $count;
}
// ========== 阶段2 ==========
function phase2() {
$total = countRecords();
if ($total <= MAX_FILE_COUNT) {
echo "阶段2:未超限,无需删除\n";
return 0;
}
$need = $total - MAX_FILE_COUNT;
$pdo = getDb();
// 修复:将 LIMIT ? 改为直接拼接数字($need 为整数,安全)
$sql = "SELECT url_md5 FROM cache_stat ORDER BY cache_time ASC LIMIT " . intval($need);
$stmt = $pdo->query($sql);
$md5List = $stmt->fetchAll(PDO::FETCH_COLUMN);
$count = 0;
foreach ($md5List as $md5) {
$file = getCacheFilePath($md5);
deleteFile($file);
deleteRecord($md5);
$count++;
}
echo "阶段2删除记录与文件数: $count\n";
return $count;
}
// ========== 阶段3(高性能批量版本) ==========
function phase3() {
$pdo = getDb();
$hexChars = ['0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f'];
$timeThreshold = time() - RETAIN_DAYS * 86400;
// ----- 第一步:生成每个二级目录的清单文件(记录数据库中的 MD5) -----
echo "[" . date('Y-m-d H:i:s') . "] 开始生成清单文件...\n";
$listFiles = []; // 记录所有生成的清单文件路径,便于后续清理
$dirBuffers = []; // 按目录分组的数据:['dirPath' => [md5, md5, ...]]
$lastMd5 = '';
$totalDbRecords = 0;
// 游标分批获取数据库记录(修正 LIMIT 拼接)
while (true) {
$sql = "SELECT url_md5 FROM cache_stat WHERE url_md5 > ? ORDER BY url_md5 LIMIT " . intval(DB_FETCH_SIZE);
$stmt = $pdo->prepare($sql);
$stmt->execute([$lastMd5]);
$rows = $stmt->fetchAll(PDO::FETCH_COLUMN);
if (empty($rows)) break;
foreach ($rows as $md5) {
// 计算所属二级目录路径(与 cache.php 规则一致)
$last1 = substr($md5, -1);
$last2 = substr($md5, -2);
$dirPath = CACHE_ROOT . DIRECTORY_SEPARATOR . $last1 . DIRECTORY_SEPARATOR . $last2;
// 按目录分组
if (!isset($dirBuffers[$dirPath])) {
$dirBuffers[$dirPath] = [];
}
$dirBuffers[$dirPath][] = $md5;
$totalDbRecords++;
$lastMd5 = $md5;
// 如果累积数据达到 FLUSH_SIZE,则写入清单文件
$totalInBuffer = array_sum(array_map('count', $dirBuffers));
if ($totalInBuffer >= FLUSH_SIZE) {
foreach ($dirBuffers as $dir => $md5List) {
$listFile = $dir . DIRECTORY_SEPARATOR . '.cache_list.txt';
file_put_contents($listFile, implode("\n", $md5List) . "\n", FILE_APPEND | LOCK_EX);
$listFiles[$listFile] = true;
}
$dirBuffers = []; // 清空缓冲区
}
}
}
// 写入剩余缓冲区数据
foreach ($dirBuffers as $dir => $md5List) {
$listFile = $dir . DIRECTORY_SEPARATOR . '.cache_list.txt';
file_put_contents($listFile, implode("\n", $md5List) . "\n", FILE_APPEND | LOCK_EX);
$listFiles[$listFile] = true;
}
echo "[" . date('Y-m-d H:i:s') . "] 清单文件生成完成,共处理数据库记录 {$totalDbRecords} 条。\n";
// ----- 第二步:遍历所有二级目录,对比清单与实际文件 -----
echo "[" . date('Y-m-d H:i:s') . "] 开始对比目录...\n";
$insertBatch = []; // 待批量插入的记录 ['md5' => $md5, 'mtime' => $mtime]
$deleteRecordBatch = []; // 待批量删除的 MD5
$deleteFileCount = 0; // 删除文件计数(直接unlink)
$processed = 0;
foreach ($hexChars as $l1) {
$dir1 = CACHE_ROOT . DIRECTORY_SEPARATOR . $l1;
if (!is_dir($dir1)) continue;
foreach ($hexChars as $l2) {
$dir2 = $dir1 . DIRECTORY_SEPARATOR . $l2 . $l1;
if (!is_dir($dir2)) continue;
// 读取该目录的清单文件
$listFile = $dir2 . DIRECTORY_SEPARATOR . '.cache_list.txt';
$listMd5s = [];
if (file_exists($listFile)) {
$content = file_get_contents($listFile);
if ($content !== false) {
$listMd5s = array_filter(explode("\n", $content), 'strlen');
}
}
$listMap = array_flip($listMd5s);
// 获取实际文件列表
$files = getPhpFiles($dir2);
$actualMd5s = [];
foreach ($files as $file) {
$md5 = basename($file, CACHE_FIX);
$actualMd5s[$md5] = $file;
}
// 找出清单中有但实际文件缺失的记录
foreach ($listMd5s as $md5) {
if (!isset($actualMd5s[$md5])) {
$deleteRecordBatch[] = $md5;
if (count($deleteRecordBatch) >= BATCH_SIZE) {
batchDeleteRecords($deleteRecordBatch);
$deleteRecordBatch = [];
usleep(SLEEP_USEC);
}
}
}
// 找出实际文件存在但清单中没有的文件
foreach ($actualMd5s as $md5 => $file) {
if (!isset($listMap[$md5])) {
$mtime = filemtime($file);
if ($mtime === false || $mtime < $timeThreshold) {
deleteFile($file);
$deleteFileCount++;
} else {
$insertBatch[] = ['md5' => $md5, 'mtime' => $mtime];
if (count($insertBatch) >= BATCH_SIZE) {
batchInsertRecords($insertBatch);
$insertBatch = [];
usleep(SLEEP_USEC);
}
}
}
$processed++;
if ($processed % 1000 == 0) {
echo "阶段3进度:已处理 {$processed} 个文件对比,累计删除文件 {$deleteFileCount}\n";
}
}
// 删除该目录的清单文件(处理完即删)
if (file_exists($listFile)) {
@unlink($listFile);
unset($listFiles[$listFile]);
}
}
}
// 处理剩余批次
if (!empty($insertBatch)) {
batchInsertRecords($insertBatch);
$insertBatch = [];
}
if (!empty($deleteRecordBatch)) {
batchDeleteRecords($deleteRecordBatch);
$deleteRecordBatch = [];
}
// 清理可能残留的清单文件
foreach ($listFiles as $file => $dummy) {
if (file_exists($file)) @unlink($file);
}
echo "阶段3完成:处理文件 {$processed} 个,删除文件 {$deleteFileCount} 个。\n";
}
// ========== 主流程 ==========
function cleanup($forcePhase3 = false) {
echo "[" . date('Y-m-d H:i:s') . "] 开始清理...\n";
phase1();
phase2();
$isThursday = (date('N') == 4);
if ($forcePhase3 || $isThursday) {
echo "[" . date('Y-m-d H:i:s') . "] 执行阶段3(全量修复)...\n";
phase3();
} else {
echo "[" . date('Y-m-d H:i:s') . "] 今日不是周四,跳过阶段3(全量修复)。\n";
}
cleanEmptyDirs(CACHE_ROOT);
echo "[" . date('Y-m-d H:i:s') . "] 清理完成。\n";
}
// ========== 脚本入口 ==========
if (php_sapi_name() === 'cli') {
ob_implicit_flush(true);
if (ob_get_level()) ob_end_flush();
$force = false;
global $argv;
if (isset($argv)) {
foreach ($argv as $arg) {
if ($arg === '--force' || $arg === '-f') {
$force = true;
break;
}
}
}
cleanup($force);
} else {
header('Content-Type: text/plain; charset=utf-8');
echo "仅允许 CLI 运行。\n";
exit(1);
}
?>V. Deployment and Scheduled Tasks
support
cache.phpPlace it in the appropriate location within the project.And ensure that it is included before all other outputs.Execute table creation SQL statement。
support
cache_cleanup.phpDeploy to the serverGrant execution authority.Add a crontab task(Executed at 4:00 AM daily):
cd /data/www/wwwroot/123.com && /usr/bin/php cache_cleanup.php
VI. Performance Optimization and Precautions
Dispersed Table of ContentsUse the last two digits of the MD5 hash value to evenly distribute files across 256 secondary directories; the number of files per directory is controllable (total number of files ÷ 256).
database index:
cache_time;access_countThe index ensures query efficiency for both Phase 1 and Phase 2.Phase III: Speed Control:
BATCH_SIZE;SLEEP_USECAdjustable to mitigate sudden I/O spikes.memory usagePhase 3: Load all MD5s into ;
$dbMapApproximately 1,000,000 records occupy 60–80 MB of disk space; ensure that the PHP memory limit is sufficiently high.exception handlingAll database operations are caught for exceptions and logged; this ensures that the script does not terminate unexpectedly.
VII. Summary
This caching system combines the performance of file caching with the flexibility of MySQL counting, enabling intelligent hotness-based cleanup. The three-phase cleanup strategy effectively evicts infrequently accessed cached data, manages total storage capacity, and provides a comprehensive recovery mechanism.
After the modification, the cache directory no longer contains unnecessary entries; pageType The unified hierarchy and path rules significantly reduce the complexity of the cleanup script. In actual deployment scenarios, even when the number of files reaches millions, Phase 1 and Phase 2 can be completed within just a few seconds; Phase 3 performs a steady, continuous scan during off-peak hours, placing minimal load on the server.
If you are also facing similar cache management challenges, you might consider adopting this approach – simply adjust the threshold parameters according to your specific requirements.