cache issue cropping up
Created 7 years ago by emergingdznsI've now got a couple clients deployed and we keep running into this problem where suddenly the cache becomes unwritable. Specifically this is the error:
[2017-10-31 15:42:11] production.ERROR: ErrorException: file_put_contents(/home/forge/beta.electronicvet.com/storage/streams/default/cache/f0/87/f0871e2f5a2c13f1d224efcc0fd587a8247e8d2e): failed to open stream: Permission denied in /home/forge/beta.electronicvet.com/vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php:111
Stack trace:
#0 [internal function]: Illuminate\Foundation\Bootstrap\HandleExceptions->handleError(2, 'file_put_conten...', '/home/forge/bet...', 111, Array)
#1 /home/forge/beta.electronicvet.com/vendor/laravel/framework/src/Illuminate/Filesystem/Filesystem.php(111): file_put_contents('/home/forge/bet...', '1510064471a:17:...', 2)
#2 /home/forge/beta.electronicvet.com/vendor/laravel/framework/src/Illuminate/Cache/FileStore.php(107): Illuminate\Filesystem\Filesystem->put('/home/forge/bet...', '1510064471a:17:...', true)
#3 /home/forge/beta.electronicvet.com/vendor/laravel/framework/src/Illuminate/Cache/Repository.php(206): Illuminate\Cache\FileStore->put('streams::module...', '1510064471a:17:...', 9999)
#4 /home/forge/beta.electronicvet.com/vendor/laravel/framework/src/Illuminate/Cache/Repository.php(316): Illuminate\Cache\Repository->put('streams::module...', Array, 9999)
#5 /home/forge/beta.electronicvet.com/vendor/laravel/framework/src/Illuminate/Cache/CacheManager.php(301): Illuminate\Cache\Repository->remember('streams::module...', 9999, Object(Closure))
#6 /home/forge/beta.electronicvet.com/vendor/anomaly/streams-platform/src/Model/EloquentModel.php(134): Illuminate\Cache\CacheManager->__call('remember', Array)
#7 /home/forge/beta.electronicvet.com/vendor/anomaly/streams-platform/src/Addon/AddonManager.php(187): Anomaly\Streams\Platform\Model\EloquentModel->cache('streams::module...', 9999, Object(Closure))
#8 /home/forge/beta.electronicvet.com/vendor/anomaly/streams-platform/src/Addon/AddonManager.php(112): Anomaly\Streams\Platform\Addon\AddonManager->getEnabledAddonNamespaces()
#9 /home/forge/beta.electronicvet.com/vendor/anomaly/streams-platform/src/StreamsServiceProvider.php(284): Anomaly\Streams\Platform\Addon\AddonManager->register()
#10 [internal function]: Anomaly\Streams\Platform\StreamsServiceProvider->Anomaly\Streams\Platform\{closure}(Object(Illuminate\Foundation\Application))
#11 /home/forge/beta.electronicvet.com/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(808): call_user_func(Object(Closure), Object(Illuminate\Foundation\Application))
#12 /home/forge/beta.electronicvet.com/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(757): Illuminate\Foundation\Application->fireAppCallbacks(Array)
#13 /home/forge/beta.electronicvet.com/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/BootProviders.php(17): Illuminate\Foundation\Application->boot()
#14 /home/forge/beta.electronicvet.com/vendor/laravel/framework/src/Illuminate/Foundation/Application.php(203): Illuminate\Foundation\Bootstrap\BootProviders->bootstrap(Object(Illuminate\Foundation\Application))
#15 /home/forge/beta.electronicvet.com/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(254): Illuminate\Foundation\Application->bootstrapWith(Array)
#16 /home/forge/beta.electronicvet.com/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(145): Illuminate\Foundation\Http\Kernel->bootstrap()
#17 /home/forge/beta.electronicvet.com/vendor/laravel/framework/src/Illuminate/Foundation/Http/Kernel.php(117): Illuminate\Foundation\Http\Kernel->sendRequestThroughRouter(Object(Illuminate\Http\Request))
#18 /home/forge/beta.electronicvet.com/public/index.php(53): Illuminate\Foundation\Http\Kernel->handle(Object(Illuminate\Http\Request))
#19 {main} {"identification":{"id":"0953728c-e9da-4dc8-8370-c9f55c165984"}} Initially, one might think that permissions got jacked up. But no, that's not the case. Nothing changed. Simply doing php artisan cache:clear fixes the problem. But the issue is this just suddenly happens and all I can do is log into the server and run the command.
Any ideas what might be causing this? It happens at least once a week and seemingly at random.
But if the directory permissions keep changing, why does a cache:clear fix it? It doesn't reset the permissions.
I went through the entire tree: storage/streams/default/cache and all of those folders are owned by forge:forge. However, inside of cache, the folders are all owned by www-data:www-data except for one which is owned by forge:forge. WTF??
Also, I think it prudent to note that no one is uploading any files at this point on these two sites. They are simply using the sites. Once one of these cache issues comes up the whole site is throwing the "Houston" error, not just that one user.
So I chown'd the folders in cache to forge:forge and then went back to the root and ran cache:clear again and now they are all owned by www-data:www-data.
I can't make heads or tails of this. Why are there two different permissions and why for the love of all that is holy would it get changed?
So the files are being created as a bad permission and then cache:clear (ran as root usually) clears it.
Right. But when I run artisan cache:clear as root, why is it setting the ownership to www-data and NOT to forge like the rest of the files? I realize this isn't a Pyro specific thing, but I appreciate any insights as to what might cause this. Thanks!
@emergingdzns I am really not sure - I think it has to do with the user that CLI is running as. If you dig into your PHP-FPM configuration it'll have the user defined there.
Do you have any scheduled tasks or anything that would be causing artisan to run when you don't explicitly do it yourself?
I had an issue before where my IMPORT scheduled command was writing to the log and next time the system had to write to log from a web request it broke cause of permissions. Seems to ONLY happen however when CLI creates the file.. If the file exists and the CLI writes to it it doesn't change permissions and web can write to it fine. However if CLI creates the log files.. then web can't write to it.
Does that make sense?
Unfortunately I'm not a strong systems administrator but our IT dude at the time fixed it 😄
I have some resources I will ask about it though for you too.
If you can - share some PHP configuration values around your user and chown behavior (usually you can set the default chown / permission level).
Usually this happens when the user creates files in a dir it doesn't own. Do you have any cron/SSH tasks that run and write storage / logs?
A lot of times it has to do with WHO creates the new file.
I would be curious why the directory (undoubtedly) keeps changing permission.