Change on Tumblr: RSS feeds not working

ontheair · May 19, 2018, 5:04pm

Hello,

this thread is not because there is a problem in Tiny Tiny RSS - Its working fine and I am using it since long years. Thank you for your great work!

This thread is about the fact that Tumblr changed something with its RSS feeds, causing problems. And maybe you can give me a hint to find a workaround.

The facts:
Tumblr-RSS feeds worked fine before. But now they made a change: When accessing any Tumblr-RSS-feed in any browser (for example: Pictures/Gifs about the Funny Gaming World.), there is a redirect, showing a page with privacy-related informations. After hitting the OK-button, the RSS-page is shown.

Because of this redirect Tiny Tiny RSS is not able anymore to read the feeds. I am searching for a way to override this OK-button, so the feeds are working again. Maybe anyone has an idea?

By the way: When checking the feed with https://fakecake.org/myfeedsucks/, it works. I dont really understand why…

Again: For sure its not the fault of Tiny Tiny RSS. It seems to be the changed behaviour of Tumblr.

My environment:
Tiny Tiny RSS v17.12 (4fa64e8)
MariaDB 10.1.23
Lighttpd 1.4.45
PHP 5.6.30
Debian Stretch

fox · May 19, 2018, 5:35pm

this is probably gdpr
are you from eu by any chance?

JustAMacUser · May 19, 2018, 6:13pm

It’s working for me; I’m not in the EU.

If you have some technical skills, see if there is anything unique about the redirect (added query strings, cookies, etc.) and then modify the feed in TT-RSS accordingly.

linoth · May 19, 2018, 7:04pm

I have a handful of Tumblr feeds as well; both normal and login gated.

As far as testing with browser, I wouldn’t recommend it. Use feed debugger (f shift+d in TTRSS) and see what happens there. Especially, as JustAMacUser pointed out, what’s going on with the redirect URLs.

In a browser, Tumblr will still force a login for an RSS feed on a gated feed. TTRSS (or CURL, etc) will still fetch the feed without any questions.

mamil · May 19, 2018, 7:12pm

Same issue as the original poster. Works fine with a US IP address, doesn’t work with a EU one. Feed debugger:

[19:05:06/7669] start
[19:05:06/7669] local cache will not be used for this feed
[19:05:06/7669] last unconditional update request: 2018-05-19 19:04:19
[19:05:06/7669] stored last modified for conditional request: 
[19:05:06/7669] fetching [http://redacted.tumblr.com/rss] (force_refetch: 1)...
[19:05:07/7669] fetch done.
[19:05:07/7669] source last modified: 
[19:05:07/7669] fetch error: LibXML error 68 at line 21 (column 29): xmlParseEntityRef: no name

[19:05:07/7669] + LibXML error 68 at line 21 (column 29): xmlParseEntityRef: no name

[19:05:07/7669] + LibXML error 68 at line 21 (column 30): xmlParseEntityRef: no name

[19:05:07/7669] + LibXML error 68 at line 21 (column 41): xmlParseEntityRef: no name

[19:05:07/7669] + LibXML error 68 at line 21 (column 42): xmlParseEntityRef: no name

[19:05:07/7669] + LibXML error 76 at line 30 (column 313): Opening and ending tag mismatch: link line 10 and head

[19:05:07/7669] + LibXML error 68 at line 38 (column 1098): StartTag: invalid element name

[19:05:07/7669] + LibXML error 68 at line 38 (column 1482): StartTag: invalid element name

[19:05:07/7669] + LibXML error 68 at line 39 (column 81): StartTag: invalid element name

[19:05:07/7669] + LibXML error 38 at line 41 (column 111): Unescaped '<' not allowed in attributes values

[19:05:07/7669] + LibXML error 65 at line 41 (column 111): attributes construct error
…

ontheair · May 19, 2018, 7:25pm

I forgot to mention. Yes, I am from EU and this problem is GDPR-related, right.

The OK-Button is not just a simple link, there is some script involved. And Im a noob in that matter… @mamil was faster and already provided the debugger log.

Ok, so this is just a problem for european users. I will try to understand how this redirect works (not very hopeful), maybe I will be able to give more informations.

Anyway, thank you for your answers already!

Edit:
When trying to access this page:
http://funnygamememes.tumblr.com/rss
Then its redirected to this page:

Not sure if you are able to see the page outside the EU.

mamil · May 19, 2018, 7:28pm

The redirect disappears once Tumblr sets a cookie. The question is how to receive a cookie on a remote VPS without a browser. Annoying as hell.

ontheair · May 19, 2018, 7:31pm

Yes, I was also thinking if its possible to catch the cookie and feed it to Tiny Tiny RSS

Is Tiny Tiny RSS using curl? And is there a chance to set a cookie for curl? Maybe thats just an weird idea, but…

mamil · May 19, 2018, 7:33pm

The GDPR page promises to vanish once you log in. I’ve signed up to Tumblr and tried to login with curl hoping that it would set the cookie, but no luck so far. Here’s what I tried.

mamil · May 19, 2018, 10:44pm

I confirm. Thanks ever so much for the tip. I’ll stick with FeedBurner for now, hoping that Tumblr will eventually get its act together and remove the GDPR screen.

fox · May 19, 2018, 11:50pm

maybe someone should contact tumblr and ask them to unfuck their rss feeds

e: we can of course also screw around with cookies and stuff

ontheair · May 20, 2018, 5:17am

maybe someone should contact tumblr and ask them to unfuck their rss feeds

I already did that before opening this thread, of course But I have little hope that they understand the problem right and I doubt that they are really caring about their feeds. Lets see.

fox · May 20, 2018, 7:34am

i find it deeply ironic that getting a vps in russia could also be solution to this issue

reibuehl · May 20, 2018, 7:46am

Is there a way to use feedburner in a “transparent” way, meaning without creating new feeds in my own account for somenody elses content?

ontheair · May 20, 2018, 11:06am

@reibuehl I dont think so. I spent some hours now, creating all the feeds (a lot!) in feedburner and telling Tiny Tiny RSS about the new (feedburner-)URLs.

Its a lot of work, but thats the way to workaround the bulls**** that Tumblr has done.

@lllusion3418 thank you for the hint!

fox · May 20, 2018, 12:59pm

wouldn’t it be easier to get a lowest tier VDS outside of EU? probably even something like openshift would work
put a simple noncaching http proxying script on it (that’s like 10 lines of code in php or w/e) and you’re done (well, limit it to tumblr rss feeds)
screwing around manually with feedburner looks like too much effort

framasky · May 20, 2018, 1:19pm

I managed to find a temporary workaround (temporary since there is an exp value with a unix timestamp in the cookie).

Get a user-agent switcher addon for your browser. Set it to use your TTRSS user-agent (should be Tiny Tiny RSS/17.12 (a7265d9) (http://tt-rss.org/) with a7265d9 being a part of the git hashsum of the latest commit of your TTRSS).
Go to a tumblr RSS URL. Accept or say “later” on the consent page
Open your browser dev tools, get the content of the pfg cookie
Apply this patch to your TTRSS (the line numbers may differ from yours as my TTRSS is patched for other purposes) and change the value of the cookie for yours:

diff --git a/include/functions.php b/include/functions.php
index 7b4747e9..fa737f65 100755
--- a/include/functions.php
+++ b/include/functions.php
@@ -383,9 +383,15 @@

                        $ch = curl_init($url);

+                       $c_headers = array();
                        if ($last_modified && !$post_query) {
-                               curl_setopt($ch, CURLOPT_HTTPHEADER,
-                                       array("If-Modified-Since: $last_modified"));
+                               array_push($c_headers, "If-Modified-Since: $last_modified");
+                       }
+                       if (preg_match('/tumblr\.com/', $url)) {
+                               array_push($c_headers, "Cookie: pfg=f1a5834399eeee1c85506760a5acbdb488884ff4cc755abc1cf5a923c182c525%23%7B%22eu_resident%22%3A1%2C%22exp%22%3A1558355685%7D%235789485527");
+                       }
+                       if (count($c_headers) > 0) {
+                               curl_setopt($ch, CURLOPT_HTTPHEADER, $c_headers);
                        }

                        curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout ? $timeout : FILE_FETCH_CONNECT_TIMEOUT);

shabble · May 20, 2018, 1:52pm

Think I’ll pass, ta.

fox · May 20, 2018, 2:01pm

+ array_push($c_headers, "Cookie: pfg=f1a5834399eeee1c85506760a5acbdb488884ff4cc755abc1cf5a923c182c525%23%7B%22eu_resident%22%3A1%2C%22exp%22%3A1558355685%7D%235789485527");

also the rest of the post, jesus wept

aeritir · May 20, 2018, 3:12pm

I hacked together a small shell script which accepts the stupid GDPR page, stores the cookie and fetches the feed afterwards. Works for me™

#!/usr/bin/env bash

DIR="$(dirname $0)"

# Base URL
FEED_URL="${1:-http://sarahcandersen.com/rss}"

# URL and payload for GDPR consent
CONSENT_URL="https://www.tumblr.com/svc/privacy/consent"
CONSENT_PAYLOAD="{ \"eu_resident\": true, \"gdpr_consent_core\": true, \"redirect_to\": \"${FEED_URL}\" }"

# Mhhhh, cookies
COOKIE_JAR="${DIR}/cookies.txt"

# Output file for the fetched feed
OUTPUT_FILE="${2:-${DIR}/feed.rss}"

curl --silent --cookie ${COOKIE_JAR} --cookie-jar ${COOKIE_JAR} \
    --request POST --header "Content-Type: application/json" \
    --data "${CONSENT_PAYLOAD}" "${CONSENT_URL}" --output /dev/null

echo "Getting feed"
curl --silent --location "${FEED_URL}" --cookie ${COOKIE_JAR} --cookie-jar ${COOKIE_JAR} --output ${OUTPUT_FILE}