Problem :
There are lot of these kinds of questions on superuser, so I’ll try and keep this quick by telling you what it’s not:
- There is no application involved here. Just a user and a filesystem.
- I can login as root and my non-root user has full sudo access.
- As far as I know, this problem only exists in one directory (though given enough time I think I can recreate this behavior in any directory — more on that below).
- No strange filesystem customizations, umasks or odd user or directory permission schemes — I own this system, and no other people interact with it.
This directory is a destination path for backup scripts written in bash. These scripts are not fancy. Rsync files from elsewhere to a working directory. Tar up all the files in the working directory and store the resulting archives in this destination path. Pretty simple stuff.
Let’s say this is a fresh install, for this example. The script works fine for a while. But after some period of weeks, no files can be created in the destination path anymore, by any user, using any method. I’ve tried touch, tar, zip, etc. Nothing works. Tar and zip will fail with I/O errors, touch acts fine but no file gets created. In that destination path, and only that path. File creation/modification/deletion works fine everywhere else on the filesystem (if it’s happening elsewhere, it’s asymptomatic).
permissions on the dir look like this via an ls -al: drwxrwxr-x. — owned by my non-root user and its group.
Lastly, if I reboot the system after this condition presents itself, I can create files in that path again for a short time. Eventually though, the problem returns.
When I said I can probably recreate this in another dir, it’s because I’ve changed the destination path to something else trying to fix this a few different times already, and it eventually happens in every location I’ve tried.
System Details:
- CentOS 2.6.32-431.3.1.el6.x86_64
- Filesystem is ext4
- 400GB free on / (lvm)
Any ideas or thoughts on things to try would be greatly appreciated. I’m happy to go into more detail if needed, but I wanted to keep this brief to start with.
Thanks for your time!
EDIT 10/20/14: Still no answers here. Can I get anyone any more info?
EDIT 10/8/14: Adding information as requested in comments:
$ ls -l
total 0
$ sudo vgs
VG #PV #LV #SN Attr VSize VFree
vg_omega1 1 2 0 wz–n- 464.24g 0
EDIT 10/7/14: Adding information as requested in comments:
Mount results:
/dev/mapper/vg_omega1-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext=”system_u:object_r:tmpfs_t:s0″)
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
Path:
/home/[username]/savegames/sync
EDIT 10/6/14: Further testing indicates I can apparently create sub-directories and symlinks just fine. This problem appears to be limited to regular files.
Solution :
I can’t comment as I don’t have the rep, but seeing your mount output it looks like you have SELinux running. From experience, it’s hard to track “weird” stuff like this down unless you’re familiar with SELinux. 5+ years dealing with it has taught me a trick or two.
tmpfs on /dev/shm type tmpfs (rw,rootcontext=”system_u:object_r:tmpfs_t:s0″)
You also mention you’re using rsync, and this problem “goes away and comes back again”. Is your SELinux options set to enforce by default? (applied again after a reboot), and you’re (sometimes) putting SELinux into permissive mode (5 mins, or 10 days etc time) after you reboot?
sudo getenforce
If the result of that is enforcing , and If you do have SELinux set to enforce by default, chances are that you’re running into SELinux policy denials.
Ref the RedHat 6 admin guide – Ch 12 – rsync
If you
ls -lZ /home/[username]/savegames/sync
What is the context label? It should be something like public_content_t
It may be user_home_t which means a lot of SELinux confined processes will be denied access.
There will be denial messages in /var/log/audit/audit.log
If SELinux is indeed the culprit, ref CentOS SELinux troubleshooting guide for ways to sort this out.
You could:
- Put SELinux into permissive mode and see if your problem goes away. Put it back into enforcing mode and your problem will come back. Not a long term solution, so…
- Either relabel the context of the /home/[username]/savegames/sync folder (to public_content_t or similar) or
- Create policies to allow your scripts/processes to work with your folder (audit2allow). This looks daunting but is not impossible.
PLEASE. DO NOT TURN SELINUX OFF.
Well, having said that, if you’re the only one using the system, and you’re not serving websites or FTP or anything, okay maybe leave it in permissive mode, but good practice suggests you modify your policy to give you a key to a door, rather than just take the door off its hinges and put it aside 🙂