Problem :
A compromised wordpress site has started several processes that run as the apache user. I can find these processes by running ps and greping for the apache user. Then I take one of the process IDs and run lsof against it to see that it has opened a bunch of tcp sockets to send mail to various recipients but I don’t see the actual file that is the root of all of this. How can I drill down further to find how this is all being started? Here is an example of one of the processes:
~# lsof | grep 26098
/sbin/ude 26098 www-data cwd DIR 252,0 4096 2 /
/sbin/ude 26098 www-data rtd DIR 252,0 4096 2 /
/sbin/ude 26098 www-data txt REG 252,0 10376 150473 /usr/bin/perl
/sbin/ude 26098 www-data mem REG 252,0 22880 143572 /usr/lib/perl/5.14.2/auto/File/Glob/Glob.so
/sbin/ude 26098 www-data mem REG 252,0 35176 143571 /usr/lib/perl/5.14.2/auto/Socket/Socket.so
/sbin/ude 26098 www-data mem REG 252,0 18632 143564 /usr/lib/perl/5.14.2/auto/IO/IO.so
/sbin/ude 26098 www-data mem REG 252,0 105720 143562 /usr/lib/perl/5.14.2/auto/POSIX/POSIX.so
/sbin/ude 26098 www-data mem REG 252,0 18600 143570 /usr/lib/perl/5.14.2/auto/Fcntl/Fcntl.so
/sbin/ude 26098 www-data mem REG 252,0 2919792 137135 /usr/lib/locale/locale-archive
/sbin/ude 26098 www-data mem REG 252,0 43288 1329383 /lib/x86_64-linux-gnu/libcrypt-2.15.so
/sbin/ude 26098 www-data mem REG 252,0 135366 1329380 /lib/x86_64-linux-gnu/libpthread-2.15.so
/sbin/ude 26098 www-data mem REG 252,0 1030512 1329394 /lib/x86_64-linux-gnu/libm-2.15.so
/sbin/ude 26098 www-data mem REG 252,0 14768 1329387 /lib/x86_64-linux-gnu/libdl-2.15.so
/sbin/ude 26098 www-data mem REG 252,0 1815224 1329389 /lib/x86_64-linux-gnu/libc-2.15.so
/sbin/ude 26098 www-data mem REG 252,0 1558296 143547 /usr/lib/libperl.so.5.14.2
/sbin/ude 26098 www-data mem REG 252,0 149280 1329381 /lib/x86_64-linux-gnu/ld-2.15.so
/sbin/ude 26098 www-data 0r CHR 1,3 0t0 5014 /dev/null
/sbin/ude 26098 www-data 1w CHR 1,3 0t0 5014 /dev/null
/sbin/ude 26098 www-data 2w CHR 1,3 0t0 5014 /dev/null
/sbin/ude 26098 www-data 3u IPv4 51672921 0t0 TCP 172.24.14.51:51017->10.81.54.194:smtp (SYN_SENT)
/sbin/ude 26098 www-data 4w FIFO 0,8 0t0 33237048 pipe
/sbin/ude 26098 www-data 5r FIFO 0,8 0t0 33237049 pipe
/sbin/ude 26098 www-data 6w FIFO 0,8 0t0 33237073 pipe
/sbin/ude 26098 www-data 7r FIFO 0,8 0t0 33237074 pipe
Solution :
Run pstree -p | less to get a full list of processes running on your system, then search for 26098 and see who its parent is, then the grandparent etc. Somewhere in that tree is your answer.
I did not manage to find a way to detect the source, but in my case the bot was discovered by clamav and was in /var/tmp directory.