Update service - feeds stop updating


#1

I had a system problem and had to do a reinstall, also, I’ve recently upgraded to F26, so there might have been a change there but:
I’ve noticed that after a period of time, I no longer get any feed updates. If I restart the ttrss-update service, they update for awhile, and then stop.
I’m running the latest git, as of this morning July 16, 2017.

Here is the output from systemctl status…
● ttrss-update.service - Tiny Tiny RSS update daemon
Loaded: loaded (/etc/systemd/system/ttrss-update.service; enabled; vendor preset: disabled)
Active: active (running) since Sat 2017-07-15 18:18:08 PDT; 16h ago
Main PID: 7104 (php)
Tasks: 62 (limit: 4915)
CGroup: /system.slice/ttrss-update.service
├─ 348 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 347
├─ 559 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 558
├─ 837 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 836
├─ 2282 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 2281
├─ 2772 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 2771
├─ 3810 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 3809
├─ 4035 /bin/php ./update_daemon2.php
├─ 4036 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 4035
├─ 4148 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 4147
├─ 4531 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 4530
├─ 5983 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 5982
├─ 6252 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 6251
├─ 6367 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 6366
├─ 7104 /bin/php ./update_daemon2.php
├─ 7266 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 7265
├─ 7500 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 7499
├─ 7915 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 7914
├─ 8266 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 8265
├─ 9347 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 9346
├─10872 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 10871
├─11071 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 11070
├─11098 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 11097
├─11101 /usr/bin/php update.php --daemon-loop --task 0 --pidlock 11099
├─12634 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 12633
├─13033 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 13032
├─13122 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 13121
├─13905 /usr/bin/php update.php --daemon-loop --task 0 --pidlock 13904
├─14158 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 14157
├─14180 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 14179
├─14557 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 14556
├─16018 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 16017
├─16158 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 16156
├─16214 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 16213
├─17489 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 17488
├─17739 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 17738
├─17805 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 17804
├─18019 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 18018
├─19176 /bin/php ./update_daemon2.php
├─19177 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 19176
├─19373 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 19372
├─19754 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 19753
├─20918 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 20917
├─20930 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 20928
├─20945 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 20944
├─22734 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 22733
├─22949 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 22948
├─23201 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 23200
├─23341 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 23340
├─24214 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 24213
├─24519 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 24518
├─25773 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 25772
├─26178 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 26177
├─26316 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 26315
├─27598 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 27597
├─27717 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 27716
├─27779 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 27778
├─27904 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 27903
├─29079 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 29078
├─29519 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 29518
├─29855 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 29854
├─30858 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 30856
└─32645 /usr/bin/php update.php --daemon-loop --task 1 --pidlock 32644

Jul 15 18:18:08 charon systemd[1]: Started Tiny Tiny RSS update daemon.
Jul 16 05:10:16 charon php[7104]: libpng warning: iCCP: known incorrect sRGB profile


#2

you can try collecting actual output (stdout) logs. as far as i can see not only the tasks themselves hang but actual job control is not working correctly either because there’s way too many update tasks running

personally i’m not interested in being an unpaid beta tester for redhat so i can’t really help you other than suggesting you use an actual server-grade distro to run server stuff on. if you prefer rpm that would be, for example, centos.


#3

Yeah, I looked at the logs and didn’t really see anything… I’ll keep looking. I most likely forgot something when I reinstalled. php only went from 7.0 to 7.1 in Fedora, so that shouldn’t have made a difference.


#4

if it’s a php 7.1 thing we’re screwed because debian 9 has 7.0 and i haven’t even updated my dev systems (still on 5.6 lol)


#5

It’s probably something I forgot to configure, but still haven’t been able to figure it out. ¯_(ツ)_/¯
I tried running from a terminal, and after about 30 minutes I start getting: [MASTER] child process 32285 seems to be stuck, aborting…
In the meantime, I just setup a cron to do it…


#6

So, it works from cron but not from systemctl? …


#7

You should probably define DAEMON_EXTENDED_DEBUG as true in your config.php file then output the logging to a file so you can see where, exactly, the script is failing.


#8

Thanks for the tip, I’ll do that and report back.


#9

No, I’m running update.php --feeds. update_daemon2.php is what fails, no matter if I run it from systemd or a terminal window. What is happening with update_daemon2.php is that initially it runs fine, everything updates - but after about 30 minutes or so the child process start to hang, and they just stack up in the lock directory. I just turned on extended debug as suggested in another comment. Hopefully, that will give a clue as to what is happening.


#10

Well, here is what I put into my config.php:
define(‘LOG_DESTINATION’, ‘syslog’);
define(‘DAEMON_EXTENDED_DEBUG’, ‘true’);

and here is the output from: journalctl -u ttrss-update.service
Jul 17 09:35:00 charon systemd[1]: Started ttrss_backend.

If anyone has any comments on how to trace this stuff, I’d appreciate it.


#11

you set tt-rss internal error logging to syslog, while your systemd unit likely outputs standard output to journald (output which has nothing to do with tt-rss error logging in the first place)

maybe you should look up how all this shit actually works because right now you’re basically pushing levers which are not connected in any way whatsoever

we already told you, define the aforementioned constant in config.php and capture daemon output

if your systemd unit which you never posted here ignores stdout or redirects it to /dev/null for whatever retarded reason just run the daemon under screen or tee or something

e: fedora users.txt

e2: well tbf i guess config.php doesn’t mention that this is strictly error logging


#12

I’ve just found this issues as well on a newly created system. Ill try and debug too and get some info.
Im running on debian 8, php 5.6.30, Tiny Tiny RSS v17.4 (c053b97).


#13

You cat use perf trace or strace for stuck processes, but if you dont know how do it, and what can you see, its unnecessary, imho.

Also something strange, cause without --quiet option update daemon should write to stdout and systemd must catch it.
Something like that:

# systemctl status tt-rss
● tt-rss.service - Tiny Tiny RSS feeds update daemon
   Loaded: loaded (/etc/systemd/system/tt-rss.service; enabled)
   Active: active (running) since Вт 2017-07-18 08:22:01 MSK; 13s ago
 Main PID: 25161 (php)
   CGroup: /system.slice/lxc.service/system.slice/tt-rss.service
           ├─25161 php /home/vhosts/rss.vhost.run/htdocs/update_daemon2.php --tasks 5
           ├─25166 php /home/vhosts/rss.vhost.run/htdocs/update_daemon2.php --tasks 5
           ├─25169 php /home/vhosts/rss.vhost.run/htdocs/update_daemon2.php --tasks 5
           ├─25170 sh -c /usr/bin/php update.php --daemon-loop   --task 3 --pidlock 25166
           ├─25173 sh -c /usr/bin/php update.php --daemon-loop   --task 4 --pidlock 25169
           ├─25174 /usr/bin/php update.php --daemon-loop --task 3 --pidlock 25166
           └─25176 /usr/bin/php update.php --daemon-loop --task 4 --pidlock 25169

июл 18 08:22:10 tt-rss update_daemon2.php[25161]: [05:22:10/25175] Feedbrowser updated, 64 feeds processed.
июл 18 08:22:11 tt-rss update_daemon2.php[25161]: [05:22:11/25175] Purged 5 orphaned posts.
июл 18 08:22:11 tt-rss update_daemon2.php[25161]: [05:22:11/25175] Removed 0 (feeds) 0 (cats) orphaned counter cache entries.
июл 18 08:22:12 tt-rss update_daemon2.php[25161]: [05:22:12/25161] [reap_children] child 25162 reaped.
июл 18 08:22:12 tt-rss update_daemon2.php[25161]: [05:22:12/25161] [SIGCHLD] jobs left: 3
июл 18 08:22:12 tt-rss update_daemon2.php[25161]: [05:22:12/25171] Scheduled 0 feeds to update...
июл 18 08:22:12 tt-rss update_daemon2.php[25161]: [05:22:12/25171] Sending digests, batch of max 15 users, headline limit = 1000
июл 18 08:22:12 tt-rss update_daemon2.php[25161]: [05:22:12/25171] All done.
июл 18 08:22:13 tt-rss update_daemon2.php[25161]: [05:22:13/25161] [reap_children] child 25164 reaped.
июл 18 08:22:13 tt-rss update_daemon2.php[25161]: [05:22:13/25161] [SIGCHLD] jobs left: 2

try to run from console(with stopped tt-rss servise) update_daemon2.php(with sudo as correct user(www-data?)). Will you see smth at stdout?

PS As for me, working fine at:

# cat /etc/debian_version 
8.8
# php -v
PHP 5.6.30-0+deb8u1 (cli) (built: Feb  8 2017 08:50:21) 
Copyright (c) 1997-2016 The PHP Group
Zend Engine v2.6.0, Copyright (c) 1998-2016 Zend Technologies
    with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2016, by Zend Technologies
# mysql --version
mysql  Ver 14.14 Distrib 5.5.55, for debian-linux-gnu (x86_64) using readline 6.3
# systemd --version
systemd 215
+PAM +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ -SECCOMP -APPARMOR

#14

Thanks for the tip about using tee, I’m running that now - just waiting for it to fail now…

I’ll look into wtf is happening to syslog after…


#15

Thanks for the reply… I’m using Fox’s suggestion to tee the output. I’ll look into what is happening with systemd… probably some default setting in Fedora-land…


#16

The define{‘DAEMON_EXTENDED_DEBUG’, ‘true’) didn’t really help. It printed out all kinds of information about what the process was doing, but when it failed, nothing more than I already knew:

[16:46:38/25321] [MASTER] spawned client 0 [PID:25324]…
[16:46:38/25326] Lock: update_daemon-25324.lock
[17:16:53/25321] [MASTER] child process 25324 seems to be stuck, aborting…
[17:16:53/25321] [reap_children] child 25324 reaped.

Yeah, the process is stuck, but why?


#17

any idea what was that PID doing before it got stuck? updating some specific feed?

also i’m not sure why it is getting killed and reaped but going by your earlier post the process actually stays. i think something is broken with your php or w/e.


#18

What if after you get a hung process you go into the database and query for the last updated feed (MAX(last_updated) or something). Once you know the feed, go to the TT-RSS web interface, select that feed and run the feed debugger f D to see how that handles it.

e: More specific query:

SELECT title FROM ttrss_feeds ORDER BY last_update_started DESC LIMIT 2;

#19

Does Fedora do SELinux? I wonder if it’s running into some bizarre permissions thing.


#20

Yes, it does, but I de-activate it.