Detecting which command/script deleting your files using inotifywait

Posted on

Problem :

One of our application servers periodically deletes file inside :

/home/test/data

The problem is we still dont know which the script or command deleting the files so we trying to use inotifywait. We are able to log datetime, but unable to retrieve required information. Can i configure/customize this inotify to show whom culprit which is deleting my files ?

Log Sample :

/opt/asd CREATE 2017/04/03-17:49:05
/opt/asd DELETE 2017/04/03-17:49:11
/opt/wira/.bash_history MODIFY 2017/04/03-17:51:29
/opt/wira/.bash_history MODIFY 2017/04/03-17:51:29
/opt/wira/.bash_history MODIFY 2017/04/03-17:51:29
/home/test/data/test DELETE 2017/04/03-17:52:16
/home/test/data/c/test DELETE 2017/04/03-17:58:00
/home/test/data/c DELETE,ISDIR 2017/04/03-17:58:00

This is our configuration

# specify log file
LOGFILE=/var/log/inotify.log
# specify target directory for monitoring
MONITOR=/home/test/data
# specify target events for monitoring ( comma separated )
# refer ro "man inotifywait" for kinds of events
EVENT=delete,modify,move

Solution :

The tool to do this is auditd which is

the userspace component to the Linux Auditing System. It’s responsible for writing audit records to the disk. Viewing the logs is done with the ausearch or aureport utilities. Configuring the audit rules is done with the auditctl utility. During startup, the rules in /etc/audit/audit.rules are read by auditctl. The audit daemon itself has some configuration options that the admin may wish to customize. They are found in the auditd.conf file.

(from the Manual). To detect a file being erased, after installing the auditd package and starting it, you audit the containing folder of the file in question, as follows:

    
    $ touch zz
    $ sudo auditctl -w /home/me -p wa
    $ rm /home/me/zz
    $ sudo cat /var/log/audit/audit.log
    type=DAEMON_START msg=audit(1491310210.803:235): auditd start, ver=2.4.5 format=raw kernel=4.8.0-45-generic auid=4294967295 pid=29913 subj=unconfined  res=success
    type=USER_AUTH msg=audit(1491310280.366:26): pid=30060 uid=1000 auid=1000 ses=2 msg='op=PAM:authentication acct="e" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/6 res=success'
    type=USER_ACCT msg=audit(1491310280.366:27): pid=30060 uid=1000 auid=1000 ses=2 msg='op=PAM:accounting acct="me" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/6 res=success'
    type=USER_CMD msg=audit(1491310280.366:28): pid=30060 uid=1000 auid=1000 ses=2 msg='cwd="/home/me" cmd=617564697463746C202D77202F686F6D652F6D6172696F202D70207761 terminal=pts/6 res=success'
    type=CRED_REFR msg=audit(1491310280.366:29): pid=30060 uid=0 auid=1000 ses=2 msg='op=PAM:setcred acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/6 res=success'
    type=USER_START msg=audit(1491310280.366:30): pid=30060 uid=0 auid=1000 ses=2 msg='op=PAM:session_open acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/6 res=success'
    type=CONFIG_CHANGE msg=audit(1491310280.390:31): auid=1000 ses=2 op="add_rule" key=(null) list=4 res=1
    type=USER_END msg=audit(1491310280.390:32): pid=30060 uid=0 auid=1000 ses=2 msg='op=PAM:session_close acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/6 res=success'
    type=CRED_DISP msg=audit(1491310280.390:33): pid=30060 uid=0 auid=1000 ses=2 msg='op=PAM:setcred acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/6 res=success'
    type=SYSCALL msg=audit(1491310299.535:34): arch=c000003e syscall=263 success=yes exit=0 a0=ffffff9c a1=21b0000 a2=0 a3=15e items=2 ppid=2441 pid=30087 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pts6 ses=2 comm="rm" exe="/bin/rm" key=(null)
    type=CWD msg=audit(1491310299.535:34): cwd="/home/me"
    type=PATH msg=audit(1491310299.535:34): item=0 name="/home/me" inode=23199747 dev=fd:00 mode=040755 ouid=1000 ogid=1000 rdev=00:00 nametype=PARENT
    type=PATH msg=audit(1491310299.535:34): item=1 name="zz" inode=23205547 dev=fd:00 mode=0100664 ouid=1000 ogid=1000 rdev=00:00 nametype=DELETE
    type=PROCTITLE msg=audit(1491310299.535:34): proctitle=726D007A7A
    type=USER_CMD msg=audit(1491310321.131:35): pid=30120 uid=1000 auid=1000 ses=2 msg='cwd="/home/me" cmd=636174202F7661722F6C6F672F61756469742F61756469742E6C6F67 terminal=pts/6 res=success'
    type=CRED_REFR msg=audit(1491310321.131:36): pid=30120 uid=0 auid=1000 ses=2 msg='op=PAM:setcred acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/6 res=success'
    type=USER_START msg=audit(1491310321.131:37): pid=30120 uid=0 auid=1000 ses=2 msg='op=PAM:session_open acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/6 res=success'

I have used the boldface for the relevant deletion event, where you can easily see the PPID, PID and user who performed the deletion.

Using inotifywait as an event trap is a good start. However, as you noted, it’s not telling you who/what, only when. Knowing when is a great way of narrowing down culprits. But, you need more info.

As a quick and (very) dirty solution, temporarily change the file permissions for your data directory, such that only a specific user or group can modify/delete files. This will result in a dump of error messages, as apps (and perhaps people) attempt to touch the files and get slapped. (Caveat: anything running as root will bypass this, so herein is my advice to run apps/jobs from service accounts, NOT root).

Those error messages will identify the process(es) attempting to delete the files, and from there you can further research which script is kicking off which process, since all child processes contain their parent process, all the way back up to init/0. Scripts will likely be an instance of bash, but their parent will either be crond (timed job) or an app.

Another possible avenue is to review your cron tables, to see which cron jobs are run immediately before the files are nuked. (If your apps run jobs without handing off to crond, look for the job schedules and logs within the apps.)

Leave a Reply

Your email address will not be published.