PDO is coming, here's what you need to know

Tiny Tiny RSS Development
1

giphy

Updated (14 Dec)

As of right now PDO has been merged into master branch. If your install is on pdo-experimental you should switch back to master now.

Updated info (10 Dec)

PDO branch is going to be merged into master soon. It is extremely likely you won’t notice any changes when this happens, things will continue on with the new database API.

If you’re running on a particularly garbage-tier hosting or distro I recommend checking if your PHP install has PDO extension enabled with support for your SQL server. Refer to your hosting or distro documentation for installing necessary packages.

Any third party plugins you might have installed may still be using using legacy database APIs. You will see entries in tt-rss system event log if they do. My advice is whining at plugin developers until they fix this and rewrite their stuff using PDO. Note that legacy wrappers are not being removed, for now (if nothing else bundled import-export plugin still uses those).

Also, no, you don’t need to do anything with your database. However, keep an eye on any SQL-related errors in tt-rss event log.

If you can’t figure out whether your server supports PDO or not you can drop this file into tt-rss directory on your server (as pdo-test.php) and open it in your browser. If there’s no errors reported, you are fine.

Old stuff

Summary

Looks like I lied when I said it would be too boring etc, we might be getting a proper database backend after all. Main code is mostly converted, there’s a few leftovers e.g. the installer and all plugins plugins are mostly converted.

PDO currently works in parallel with legacy database connection, not sure yet how this is going to be handled when this is merged into master. Legacy connection is established if requested, stack trace for the first attempt is logged.

Since this is a huge, invasive change I’m going to post a separate unstable branch on Gogs which would be available for testing to everyone until we’re all satisfied with stability.

I wouldn’t recommend using it as a daily driver at the moment but I would appreciate everyone checking it out and clicking on things until you find bugs and/or SQL-related warnings in the logs. Click on everything, import your OPML, see if all feeds look right, etc etc etc.

Unstable branch is available here: https://git.tt-rss.org/git/tt-rss/src/pdo-experimental

It took a lot of effort to go through literally all SQL queries in tt-rss core to make this happen. Feedback would be appreciated.

3

I submitted one bug fix for the API.

I also found another bug. It seems search_to_sql() is mishandling some searches. If I do something like title:Apple as a search parameter it fails. It seems the PDO::quote() is quoting the search term but you’ve already wrapped that text in quotes. Other searches with commands (note, etc.) are handled similarly but see line 1388 in functions.php:

array_push($query_keywords, "($not (LOWER(ttrss_entries.title) LIKE '%".
    $pdo->quote(mb_strtolower($commandpair[1]))."%'))");

It seems the % should be included with $commandpair[1] as otherwise the query ends up looking like:

LOWER(ttrss_entries.title) LIKE '%'Apple'%'

Which fails because then Apple is not part of the LIKE clause.

4

yeah you’re right, % should be added first and the resulting string quoted i guess.

e: search should be unfucked on pgsql now, idk about mysql

5

Seems to work on MySQL as well. Thanks.

6

btw in case anyone is wondering i’m literally dogfooding this atm and it seems to work

7

It works for me. But I don’t do anything fancy. Postgresql backend by the way.

8

As I type, Tiny Tiny RSS v17.12 (0518510), ubuntu server 16.04.3 LTS, postgresql, php7.0… physical machine, new tt-rss pdo-exp install, new database install.

OPML importing (44 feeds, 6 custom categories + default ‘Uncategorized’) is failing (‘power user profile’). Nothing gets imported. admin’s Preference|System|Error Log shows errors like this each time an import is attempted:

E_WARNING (2) classes/opml.php:579 DOMDocument::load(): XML declaration allowed only at the start of the document in /var/www/tt-rss/cache/upload/opmliveDw9, line: 31. classes/opml.php(579): load(/var/www/tt-rss/cache/upload/opmliveDw9)
2. classes/dlg.php(25): opml_import(2)
3. backend.php(123): importOpml()
marty 8:26

E_WARNING (2) classes/opml.php:579 DOMDocument::load(): XML declaration allowed only at the start of the document in /var/www/tt-rss/cache/upload/opmlasmSML, line: 31. classes/opml.php(579): load(/var/www/tt-rss/cache/upload/opmlasmSML)
2. classes/dlg.php(25): opml_import(2)
3. backend.php(123): importOpml()
marty 1:26’

And the popup:

opml_import_wtf
opml_import_wtf1024×695 104 KB

Checked ownership (www-data) and permissions on all files and folders and all looks good. At the moment I’ve manually entered 9 feeds. Those 9 feeds are updating as expected.

New unrelated to above (i think?) errors with the latest pull (0518510) this a.m. (CST) … log errors which I think are generated when going into ‘Actions|Preference’ ? Errors that look like this:

E_WARNING (2) include/functions.php:2368 file_get_contents(js/deprecated.js): failed to open stream: No such file or directory1. include/functions.php(2368): file_get_contents(js/deprecated.js)
2. prefs.php(95): get_minified_js(Array)
admin 10:07
E_WARNING (2) include/functions.php:2354 filemtime(): stat failed for js/deprecated.js1. include/functions.php(2354): filemtime(js/deprecated.js)
2. prefs.php(95): get_minified_js(Array)
admin 10:07
E_WARNING (2) include/functions.php:2368 file_get_contents(js/deprecated.js): failed to open stream: No such file or directory1. include/functions.php(2368): file_get_contents(js/deprecated.js)
2. prefs.php(95): get_minified_js(Array)
marty 9:19
E_WARNING (2) include/functions.php:2354 filemtime(): stat failed for js/deprecated.js1. include/functions.php(2354): filemtime(js/deprecated.js)
2. prefs.php(95): get_minified_js(Array)
marty 9:19

Oh, also. right-click context menu on a headline… ‘Assign label’ / ‘Remove label’ options are missing.

That’s about all I’ve notice so far.
,

9

deprecated.js is nbd, it’s just a file that got removed.

i’m not sure why doesn’t your uploaded OPML parse properly, this isn’t really something tt-rss is responsible for (php handles the upload and DOMDocument parses the xml). i’ll make a note to try importing opml later.

labels are likely missing because you don’t have any labels.

10

Crap! That was in fact reason for the missing ‘Assign label’ / ‘Remove label’ options . I created a label, now the options are there in the context menu. Sorry.
.

11

Ok, thanks for the hint. Turns out the exported OPML file is saved with a couple of lines white space at the top which was causing the import to immediately choke. Removed the two lines of white space and then did the import again. 44 feeds listed in the popup, but further choking occurred at the end of the import process which seemed to have something to do with filters I had configure. So, did another export with the Export OPML ‘include settings’ option un-ticked. Import successful!

Thanks again for the help.
.

12

that’s strange, so if you export with settings OPML is broken? i’ll take a look, thanks.

13

i tried both settings and no settings and it seems to work fine for me here. could be empty lines outside of <?php ... ?> tags somewhere (config.php?).

14

wtf? No, no empty lines that I can find outside the tag <?php ... ?> in 'config.php. Nor in ‘opml.php’ or ‘dlg.php’ or ‘backend.php’ mentioned in the first log output posted above.

The ‘empty lines’ causing the issue (for me) are in the OPML export file. I’ve tried exporting the OMPL file in various browsers (in linux) with always the same result, two empty lines at the top of the export file (shown in red below):

whitespace_begin_opml_export_file

Once I delete those two lines so that ‘<?xml …’ is on the first line with no space before the tag and saved, then importing is successful. Well successful if… (see below)

I found through further experimenting that I can save the export file with the ‘include settings’ option enabled. … If…, I first edit out any ‘filter’ data towards the bottom of the OPML export file. With the filter data edited out, importing to tt-rss pdo-experimental succeeds.

idk? If no one else is seeing this issue don’t waste anymore of you time. Obviously.

.

15

What you’re describing is textbook white space outside of PHP tags, as fox mentioned. Thing is, it could be anywhere in your installation: config.php, third-party plugin, etc. Browsers are typically forgiving of this kind of thing, which is why your install probably works. But the OPML file parsing (on import) is entirely unforgiving.

You’ll also see white space after the PHP tags show up as “headers already sent” a lot of times when there are some types of errors early on in the script execution. But this might not be obvious since in a production environment you’d probably have disabled display_errors.

If you have third-party plugins, try disabling them and seeing it if happens. If so, double-check the config.php file (you don’t even need the closing ?> tag in that file, just leave it out). You can also do a git checkout -- . in the tt-rss directory and that will put all the core files back to their proper state for the commit you’re at (leaving only config.php, themes.local, and plugins.local to check).

16

ah so this is on your old instance?

also ^ what @JustAMacUser said

e: since we’re discussing PDO branch here try exporting/importing OPML on it

17

Ah yeah. Old (v17.4) export to new pdo-exp.

As per @JustAMacUser 's suggestion, I turned off all plugins in (v17.12).

Ok. Did another export from v17.12 with plugins disabled. This time no empty lines at the top of the export file. But, on import to another newly created user profile (in v17.12) the import still gets jammed up at the point where filters are being read.

It’s getting late here in my part of the world. I’ll play with this some more tomorrow.
.

18

interesting. just tried and it worked (exported on pgsql, imported on mysql). check for any logged SQL errors later.

19

Been using it for a day, and with the exception of forgetting that I had af_fullpost installed on the old version and not on the new, have not come across any obvious issues.

20

Heads up: I think I’m going to merge PDO branch in a few days unless something comes up. Again, if you are capable of switching branches now please consider doing so.