Getting a NodeBB RSS feed requiring authentication


#1

PLEASE READ THIS BEFORE POSTING: About the Support category

That URL doesn’t work - may want to update it and/or pin it to the top of https://discourse.tt-rss.org/c/tiny-tiny-rss/support

Describe the problem you’re having:

I have a feed (https://what.thedailywtf.com/category/4.rss for what it’s worth - though it’s a staff area. https://what.thedailywtf.com/category/49.rss is similar in that it requires you to be logged in, but no other privileges, if anyone’s interested in looking into this) that requires a login before being able to read.

Board software being used is NodeBB.

Naively setting the following (and after some thought - it naturally) doesn’t work:

Basically because without the login credentials expected, it 30x redirects to the login page.

If possible include steps to reproduce the problem:

Nothing beyond what’s in that screenshot and trying a feed that requires a login.

tt-rss version (including git commit id):

-bash-4.1$ git rev-parse --short HEAD
6fd0399

Platform (i.e. Linux distro, PHP, PostgreSQL, etc) versions:

-bash-4.1$ uname -a
Linux s15474768.onlinehome-server.info 2.6.32-042stab120.5 #1 SMP Tue Oct 25 22:31:12 MSK 2016 x86_64 x86_64 x86_64 GNU/Linux
-bash-4.1$ cat /etc/issue
CentOS release 6.8 (Final)
Kernel \r on an \m
-bash-4.1$ php --version
PHP 5.5.38 (cli) (built: Oct 15 2016 10:01:32) 
Copyright (c) 1997-2015 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2015 Zend Technologies
-bash-4.1$ mysql --version
mysql  Ver 14.14 Distrib 5.5.53, for Linux (x86_64) using readline 5.1

Please provide any additional information below:

Just wondering what workarounds might exist.


#2

the normal procedure for protected feeds is either http auth (which doesn’t work here) or adding some unique hidden token to the url (for example like reddit does it). since your forum supports neither and expecting an rss reader to somehow login to the forum is ridiculous, i guess they might as well remove those feeds because they are useless.

a workaround would be whoever developed this forum getting a brain.


#3

I’ll file a report with them. Cheers.


#4

The only way would be a script using curl or wget, which authenticates and pulls the feed somewhere locally on your TT-RSS server where it can be read but that’s a very bad workaround. As fox said, this should be fixed by the software developers.


#5

I think I’ll leave Rube Goldberg alone for this one…


#6

If you still want to try it as described, I hacked together a small shell script using curl.
It seems that Node BB requires both a valid session cookie and a token. Took me a while to figure out the correct procedure.

Just run it regularly using cron and place the output file in a web folder where your TT-RSS can fetch it (like http://yourdomain.xyz/myfeeds/wtf-feed.rss).


#!/usr/bin/env bash

USERNAME="your_user"
PASSWORD="your_pass"

# Base URL
BASE_URL="https://what.thedailywtf.com"

# URLs for login and REST API status
LOGIN_URL="${BASE_URL%/}/login"
API_URL="${BASE_URL%/}/api/config"

# URL for feed
FEED_URL="${BASE_URL%/}/category/49.rss"

# Mhhhh, cookies
COOKIE_JAR="${HOME}/cookies.txt"

# Output file for the fetched feed
OUTPUT_FILE="${HOME}/wtf-feed.rss"


# Get some REST API information
echo "Querying REST API"
API_INFO="$(curl --silent "${API_URL}" -b ${COOKIE_JAR} -c ${COOKIE_JAR})"

# Very crude JSON parsing using regular expressions :D
LOGIN_STATUS="$(echo "${API_INFO}" | grep -Po '"loggedIn":\K[^,]+')"
TOKEN="$(echo "${API_INFO}" | grep -Po '"csrf_token":"\K[^"]+')"

# Login if our previous session is no longer valid
if [[ ${LOGIN_STATUS} != true ]]; then
    echo "Logging in..."
    curl -X POST --silent "${LOGIN_URL}" -b ${COOKIE_JAR} -c ${COOKIE_JAR} \
    -d "username=${USERNAME}&password=${PASSWORD}" \
    -H "X-CSRF-Token: ${TOKEN}" > /dev/null
fi

echo "Getting feed"
curl --silent "${FEED_URL}" -b ${COOKIE_JAR} -c ${COOKIE_JAR} -H "X-CSRF-Token: ${TOKEN}" -o ${OUTPUT_FILE}