Last commit broke some feeds


#1

Commit b14f6d58b4 broke some feeds.

Example: https://pastebin.com/sX0Nz8Dd
However it works when I increase CURLOPT_BUFFERSIZE - https://pastebin.com/bs4ctetD


#2

What did you adjust CURLOPT_BUFFERSIZE to? I am seeing the same errors (unable to fetch: 28 Operation timed out after 15001 milliseconds) on a few feeds after updating to b14f6d5 this evening. The failing feeds work in CURL and https://fakecake.org/myfeedsucks/.


#3

512 works just fine for me. This is a random value to just confirm the issue.


#4

it works fine here with your feed url (https://habr.com/rss/hubs/all/) on debian 9

which platform are you using? also, file a PR i guess


#5

Ubuntu 16.04.4 LTS
PHP 7.0.30-0ubuntu0.16.04.1
curl 7.47.0

Sorry, “PR” is … -?


#6

strange, should be similar enough to debian. maybe it’s network-related somehow.

pull request with your change

e: if you don’t know how to do that don’t bother, i’ll update this later today


#7

I’m on Ubuntu 16.04.4 LTS. I am seeing a couple of the feeds that have been constantly failing occasionally complete a refresh, a couple being https://tt-rss.org/feeds/tt-rss.xml and https://www.reddit.com/r/cats/.rss.


#8

Ah, I see…


#9

btw i wonder if it would be a good idea to increase this further, to something like 8192 bytes. we don’t really need the granularity and it would probably improve performance.


#10

To be honest, I have no idea.
256 works just fine for me, it fixes the issue.


#11

Just did a few tests…
It’s possible to fix the issue by increasing the timeout (used 30 sec instead of 15).

CURLOPT_PROGRESSFUNCTION is too slow, I guess.

So, yes, it’s a good idea to increase the size of the buffer even further.


#12

whoa. what kind of hardware are we talking about here?

i would prefer to keep the timeout as-is, increasing the buffer size seems like a better idea. maybe something like 16K.


#13

VPS at DigitalOcean

configuration: boot=normal cpus=1 family=DigitalOcean_Droplet uuid=8072F01B-E808-4D84-B47D-67139CA0AB52
 *-cpu
      description: CPU
      product: Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz
      bus info: cpu@0
      slot: CPU 1

#14

looks like writefunction is not affected by buffer size btw:

not sure about progress function

anyway, i’ve increased the buffer to 16384 bytes.

looks like progress function is still called quite a lot, even with a larger buffer size:

[07:44:48/28886] [curl progressfunction] 0 16777216
[07:44:48/28886] [curl progressfunction] 0 16777216
[07:44:48/28886] [curl progressfunction] 0 16777216
[07:44:48/28886] [curl progressfunction] 0 16777216
[07:44:48/28886] [curl progressfunction] 0 16777216
[07:44:49/28886] [curl progressfunction] 0 16777216
[07:44:49/28886] [curl progressfunction] 0 16777216
[07:44:49/28886] [curl progressfunction] 0 16777216
[07:44:49/28886] [curl progressfunction] 0 16777216
[07:44:49/28886] [curl progressfunction] 0 16777216
[07:44:49/28886] [curl progressfunction] 0 16777216
[07:44:49/28886] [curl progressfunction] 0 16777216
[07:44:49/28886] [curl progressfunction] 0 16777216
[07:44:49/28886] [curl progressfunction] 0 16777216
[07:44:49/28886] [curl progressfunction] 0 16777216
[07:44:49/28886] [curl progressfunction] 0 16777216
[07:44:49/28886] [curl progressfunction] 15853 16777216
[07:44:49/28886] [curl progressfunction] 15853 16777216
[07:44:49/28886] [curl progressfunction] 15853 16777216
[07:44:49/28886] [curl progressfunction] 15853 16777216
[07:44:49/28886] [curl progressfunction] 15853 16777216
[07:44:49/28886] [curl progressfunction] 15853 16777216
[07:44:49/28886] [curl progressfunction] 15853 16777216
[07:44:49/28886] [curl progressfunction] 15853 16777216
[07:44:49/28886] [curl progressfunction] 15853 16777216
[07:44:49/28886] [curl progressfunction] 15853 16777216
[07:44:49/28886] [curl progressfunction] 32229 16777216
[07:44:49/28886] [curl progressfunction] 32229 16777216
[07:44:49/28886] [curl progressfunction] 37188 16777216
[07:44:49/28886] [curl progressfunction] 37188 16777216
[07:44:49/28886] [curl progressfunction] 37188 16777216
[07:44:49/28886] [curl progressfunction] 37188 16777216

etc etc


#15

I had been testing increased timeouts since it seemed to be a timeout issue based on the error messages I saw in the GUI. I removed all that from and added MAX_DOWNLOAD_FILE_SIZE = 0 to config.php per your recommendation and will monitor it today.


#16

You can try to turn off curl. Define NO_CURL in config.php.


#17

you can also define MAX_DOWNLOAD_FILE_SIZE as 0 in config.php which should disable max size checking and progress function

also: try increasing block size, instead of timeouts.


#18

I removed any extra declarations from config.php and performed a git update yesterday morning. Everything’s been working fine.