No space left on device (/var/lib/psa/dumps)

[Eric Pretorious]

We started getting phone calls from clients complaining that they couldn't send/receive e-mail today so we checked the maillog and found dozens of these messages:

Oct 18 17:26:19 localhost postfix/smtpd[12616]: connect from atl4mhfb01.myregisteredsite.com[209.17.115.55]
Oct 18 17:26:21 localhost postfix/smtpd[12616]: NOQUEUE: reject: MAIL from atl4mhfb01.myregisteredsite.com[209.17.115.55]: 452 4.3.1 [COLOR="#FF0000"]Insufficient system storage[/COLOR]; proto=ESMTP helo=<atl4mhfb01.myregisteredsite.com>
Oct 18 17:26:21 localhost postfix/smtpd[12616]: [COLOR="#FF0000"]warning: not enough free space in mail queue[/COLOR]: 0 bytes < 1.5*message size limit
Oct 18 17:26:21 localhost postfix/cleanup[12792]: 5BB8C11D146: message-id=<20131018172621.5BB8C11D146@localhost.example.com>
Oct 18 17:26:21 localhost postfix/cleanup[12792]: warning: 5BB8C11D146: write queue file: [COLOR="#FF0000"]No space left on device[/COLOR]
Oct 18 17:26:21 localhost /usr/lib64/plesk-9.0/psa-pc-remote[1161]: Message aborted.
Oct 18 17:26:21 localhost postfix/smtpd[12616]: disconnect from atl4mhfb01.myregisteredsite.com[209.17.115.55]

Oct 18 17:26:58 localhost pop3d-ssl: Connection, ip=[::ffff:64.160.116.241]
Oct 18 17:27:00 localhost pop3d-ssl: IMAP connect from @ [::ffff:64.160.116.241]INFO: LOGIN, user=ashley@example.com, ip=[::ffff:64.160.116.241]
Oct 18 17:27:00 localhost pop3d-ssl: [COLOR="#FF0000"]Error while saving courierpop3dsizelist[/COLOR], user=ashley@example.com

Oct 18 17:27:02 localhost imapd: Failed to create cache file: maildir_lock (rachel@example.com)
Oct 18 17:27:02 localhost imapd: [COLOR="#FF0000"]Error: No space left on device[/COLOR]
Oct 18 17:27:02 localhost imapd: Failed to create cache file: maildir_lock (rachel@example.com)
Oct 18 17:27:02 localhost imapd: [COLOR="#FF0000"]Error: No space left on device[/COLOR]
Oct 18 17:27:02 localhost imapd: Failed to create cache file: maildir_lock (rachel@example.com)
Oct 18 17:27:02 localhost imapd: [COLOR="#FF0000"]Error: No space left on device[/COLOR]

We began looking at file system usage and found that /var/lib/psa/dumps contained nearly 10,000 MySQL dump files (that consumed 43GB of file system):

[root@www log]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1             50G   48G     0 100% /
none                  497M     0  497M   0% /dev/shm

[root@www log]# du --max-depth=1 --bytes / | sort -n | tail -4
618903209	/lib
1301514097	/usr
47690668924	/var
49934886076	/

[root@www log]# du --max-depth=1 --bytes /var/lib | sort -n | tail -4
52998144	/var/lib/rpm
94939032	/var/lib/mysql
43721802237	/var/lib/psa
43886153510	/var/lib

[root@www log]# find /var/lib/psa/dumps/ -type f | wc -l
9806

So we immediate got rid of any dump files older than one month:

[root@www log]# find /var/lib/psa/dumps/ -type f -ctime +30 -ls | wc -l
68

[root@www log]# find /var/lib/psa/dumps/ -type f -ctime +30 -exec /bin/rm {} \;

[root@www log]# find /var/lib/psa/dumps/ -type f -ctime +30 -ls | wc -l
0

[root@www log]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1             50G   48G     0 100% /
none                  497M     0  497M   0% /dev/shm

Then we got a bit more aggressive and got rid of any dump files older than two weeks:

[root@www log]# find /var/lib/psa/dumps/ -type f -ctime +15 -ls | wc -l
1161

[root@www log]# find /var/lib/psa/dumps/ -type f -ctime +15 -exec /bin/rm {} \;

[root@www log]# find /var/lib/psa/dumps/ -type f -ctime +15 -ls | wc -l
0

[root@www log]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda1             50G   42G  5.2G  90% /
none                  497M     0  497M   0% /dev/shm

Removing the dump files older than two weeks has allowed us to continue operating but there are still quite a few dump files left over:

[root@www log]# find /var/lib/psa/dumps/ -type f | wc -l
8577

[root@www log]# du --max-depth=1 --bytes /var/lib/psa | sort -n | tail -3
37700314173	/var/lib/psa/dumps
37700318269	/var/lib/psa

1. What are these files?
2. Why are approximately 90% of them (i.e., 8,577 of the 9,806 dump files) less than two weeks old?
3. Is this normal?
4. Has there been a change in Plesk Panel that may have caused this?
5. Is there an automated tool for managing these?

 

[Eric Pretorious]

FWIW: The files have names like...

[root@www dumps]# find /var/lib/psa/dumps/ -type f -name '[COLOR="#FF0000"]backup_file_sharing_*[/COLOR]' | wc -l
16
[root@www dumps]# du --bytes --total backup_file_sharing_* | tail -1
4,688	total

[root@www dumps]# find /var/lib/psa/dumps/ -type f -name '[COLOR="#FF0000"]backup_keyXX*[/COLOR]' | wc -l
15
[root@www dumps]# du --bytes --total backup_keyXX* | tail -1
58,109	total

[root@www dumps]# find /var/lib/psa/dumps/ -type f -name '[COLOR="#FF0000"]backup_sapp-distrib.*[/COLOR]' | wc -l
45
[root@www dumps]# du --bytes --total backup_sapp-distrib.* | tail -1
578,640,480		total


[root@www dumps]# find /var/lib/psa/dumps/ -type f -name '[COLOR="#FF0000"]backup_sb_server_*[/COLOR]' | wc -l
15
[root@www dumps]# du --bytes --total backup_sb_server_* | tail -1
22,181	total

[root@www dumps]# find /var/lib/psa/dumps/ -type f -name '[COLOR="#FF0000"]backup_skel_*[/COLOR]' | wc -l
16
[root@www dumps]# du --bytes --total backup_skel_* | tail -1
1,496,976	total

[root@www dumps]# find /var/lib/psa/dumps/ -type f -name '[COLOR="#FF0000"]mysql.daily.dump*[/COLOR]' | wc -l
9
[root@www dumps]# du --bytes --total mysql.daily.dump* | tail -1
6,013,661	total

Though the subdirectories (i.e., clients, resellers, and domains) contain quite a few too:

./clients
7,722 files
32,959,452 bytes

./resellers
272 files
549,440 bytes

./domains
299 files
2,811,200 bytes

 

[Eric Pretorious]

1. What are these files?
2. Why are approximately 90% of them (i.e., 8,577 of the 9,806 dump files) less than two weeks old?
3. Is this normal?
4. Has there been a change in Plesk Panel that may have caused this?
5. Is there an automated tool for managing these?

Though the subdirectories (i.e., clients, resellers, and domains) contain quite a few too:

[COLOR="#FF0000"]./clients
7,722 files
32,959,452 bytes[/COLOR]

./resellers
272 files
549,440 bytes

./domains
299 files
2,811,200 bytes

It didn't take very long for me to realize the root cause of this log jam: Two weeks ago I'd installed+enabled system-wide backups (Tools & Settings -> Tools & Resources: Backup Manager) without realizing that some subscriptions would be backed up twice -- once by the system-wide settings and once by the clients' settings.

I've since figured out to review the psa database (localhost:3306 » psa » BackupsScheduled) to see which clients have scheduled backups just to keep things manageable.

 

[Eric Pretorious]

I've since figured out to review the psa database (localhost:3306 » psa » BackupsScheduled) to see which clients have scheduled backups just to keep things manageable.

I've deleted a few customers but noticed that their backups still exist in /var/lib/psa/dumps/clients.

Is there a tool/feature to remove the backups of clients that have been deleted?

 

[Eric Pretorious]

I've since figured out to review the psa database (localhost:3306 » psa » BackupsScheduled) to see which clients have scheduled backups just to keep things manageable.

FTR: The obj_id column of localhost:3306 » psa » BackupsScheduled correlates directly with the id column of localhost:3306 » psa » domains.

HTH,