Yesterda...as cleaning up carbon (a database server) and was attempting to remove files fro...ounted raid device that ha...lone of the live file system. Cleaning involves removing unused/unneeded directorie...iles. Well the location I'm cleaning ha...opy o...o naturally there are directories and files in there like: /usr, /etc/, /var and others. The cleaning involves my favourite command rm -f...hich can be dangerous. Such as the following sequence:# cd /mnt/raid/root-copy/# rm -f...etc# rm -f...usr# rm -fr /varShit! Forgot th...refix on that last command...it CTRL+C as fast a...ould! Time to see wha...illed!The most important thing was my database in /var/lib/postgresql. S...as very angry at myself when this happend:# stat /var/lib/postgresqlstat: cannot stat `/var/lib/postgresql': No such file or directoryshit shit shitThere went my database! All my client records, histories, book-keepin...ccounting back to 2004. My last conscious backup was from the first of the mont...5 days ago. Fortunatel...ad an automated backup script running that had yesterdays data. But what about all the wor...id today...ate repeating myself. Well, here'...ool trick. On UNIX style OSes the files are not actually deleted when you say rm. They are simply marked as deleted, only in the inode structure of the file-system. Once all processes close that file then the inode structure (with reference count zero) says's that space on the disc is free and can then be used. S...ad delete my /var/lib/postgresql directory but guess what. PostgreSQL had 100s of open files in there. Hooray. And the postgres process still had access to all those files in that directory!So...ooked at the process and it's open files (/proc/[pid]/fd/) and could se...ot of open files. Hoping for the bes...e-ran my postgres backup script. It had full access to all data, in all databases (14) and dumped out my 2.5GiB worth of book-keeping and company financial records. There is an interesting document on the proc file-descriptors and undeleting from finalco...he...topped the PostgreSQL server it was dead, wouldn't restart and had to have it's whole data area re-initialized...as able to recover the system right back to the point (4am) whe...ad issued the errant rm -fr command. For the recor...now that rm -fr is dangerous...requently find myself telling others to be careful with it. Remember rm -fr is short hand for rm --fuck-it --reallySo ye...ad backups, but they were more than 12 hours old, who wants that. Thanks to open file-descriptor...as very very lucky and able to recover the data to point-in-time of failure. BACKUP YOUR DATA!!!http://blog. edoceo.com/
