Feature Request: Support array of URLs on the extract.php endpoint

Feed Creator
1

Let’s say I need to retrieve text content from 100 different web pages, instead of submitting 100 different requests to my Full-text RSS installation, I could submit them in batches of 10, which would significantly reduce the amount of work my client app needs to do.

We could simply submit a JSONified array of URL/link strings in a POST HTTP request.

Is this something you think you would want to support?
Or is this something that is already possible, but I may have completely missed? (sorry, if that’s the case)

Thanks!

2

Hi Samuel,

It’s not something that we support in Full-Text RSS, but the makefulltextfeed.php endpoint can work with multiple URLs, and even fetch them in parallel, but you’ll have to give it a feed URL. So one way to give it an arbitrary list of URLs is to first create a feed from those URLs and then use the makefulltextfeed.php endpoint. Our Feed Creator allows you to do this - see http://createfeed.fivefilters.org/index-fromlist.php

I’m not sure if this is something we’ll support directly in Full-Text RSS. We haven’t ruled it out, but it’s also not a big priority.

Hope that’s some help.

Best, Keyvan from FiveFilters.org

3

Hi Keyvan,

Thanks for your response, and, thanks for the workaround. Much appreciated. :slight_smile:
Could you please also point me to where I might find some documentation on how to use the index-fromlist.php/urls.php endpoint of the Feed Creator tool, in code? This looks very close to what I was after, but I would like to be able to do this in code, instead of manually via a web browser.

Thanks again!

Best,

Samuel

Samuel J.

4

Hi Samuel,

There’s no documentation yet, but to do it in code you can simply use urls.php as the endpoint and construct the request URL as follows:

urls.php?url[]=URL1&url[]=URL2&url[]=URL3

There’s an optional title element you can pass via the t parameter:

urls.php?t=title&url[]=URL1&url[]=URL2&url[]=URL3

urls.php returns a very simple RSS feed with the URLs you’ve given it - it makes no request for any external content. To use it with Full-Text RSS, you pass the constructed URL to Full-Text RSS with the following parameter:

use_extracted_title=1

The above tells Full-Text RSS to place each title it extracts from the URLs into the feed it generates (by default the item titles in the source feed are preserved in the output feed, which in this case would mean no titles at all).

So your final URL would look something like:

…makefulltextfeed.php?use_extracted_title=1&format=json&url=…urls.php%3Furl%5B%5D%3DURL1%26url%5B%5D%3DURL2%26url%5B%5D%3DURL3

Hope that’s some help. Let me know if anything’s not clear.

Best, Keyvan from FiveFilters.org