Commit Graph

239 Commits

Author SHA1 Message Date
Frédéric Guillot 7d278d49f1 Add content length check when refreshing feeds 2018-01-04 18:41:23 -08:00
Frédéric Guillot efac11e082 Handle more date formats 2018-01-03 18:59:29 -08:00
Frédéric Guillot ec63cbe7bb If the website URL is empty, assign the feed URL 2018-01-03 18:23:21 -08:00
Frédéric Guillot c39f2e1a8d Rename helper packages 2018-01-02 19:15:08 -08:00
Frédéric Guillot 3c3f397bf5 Make sure the scraper parse only HTML documents 2018-01-02 18:32:01 -08:00
Frédéric Guillot c454f67037 Add scraper rules for version2.dk and ing.dk 2017-12-27 19:44:23 -08:00
Frédéric Guillot d4839b5597 Add more scraper rules 2017-12-27 13:36:07 -08:00
Frédéric Guillot f6a5d7d6ed Add support for data URL favicons 2017-12-22 19:01:39 -08:00
Frédéric Guillot e7afec7eca Handle more date formats 2017-12-22 17:59:28 -08:00
Frédéric Guillot 1d8193b892 Add logger 2017-12-15 18:55:57 -08:00
Frédéric Guillot c6d9eb3614 Improve content scraper 2017-12-13 21:30:40 -08:00
Frédéric Guillot 827683ab59 Make sure that item URL are absolute 2017-12-13 20:16:15 -08:00
Frédéric Guillot 84d912c979 Rewrite imports 2017-12-12 21:48:13 -08:00
Frédéric Guillot ef097f02fe Add the possibility to enable crawler for feeds 2017-12-12 19:19:36 -08:00
Frédéric Guillot 33445e5b68 Add the possibility to define rewrite rules for each feed 2017-12-11 22:16:32 -08:00
Frédéric Guillot 87ccad5c7f Add scraper rules 2017-12-10 20:51:04 -08:00
Frédéric Guillot 7a35c58f53 Add readability package to fetch original content 2017-12-10 19:01:38 -08:00
Frédéric Guillot 6f5350a497 Move packages http and url 2017-12-02 20:26:21 -08:00
Frédéric Guillot 2356ddad28 Add Pinboard integration 2017-12-02 19:32:14 -08:00
Frédéric Guillot fb2a73c91e Proxify image enclosures 2017-12-01 22:29:18 -08:00
Frédéric Guillot bb8e61c7c5 Make sure golint pass on the code base 2017-11-27 21:40:05 -08:00
Frédéric Guillot bd663b43a0 Improve HTML sanitizer 2017-11-25 18:08:59 -08:00
Frédéric Guillot 71bf7e4358 Improve API 2017-11-24 22:29:20 -08:00
Frédéric Guillot 2b641cc224 Improve feed parsers 2017-11-22 14:52:31 -08:00
Frédéric Guillot 99dfbdbb47 Convert feed encoding only if the charset is specified 2017-11-21 22:55:19 -08:00
Frédéric Guillot 5f0ae8196c Add timeout for HTTP client 2017-11-20 19:44:28 -08:00
Frédéric Guillot eb9f588216 Make sure RDF entries have a date 2017-11-20 19:25:30 -08:00
Frédéric Guillot d5838b6734 Move feed parsers packages in reader package 2017-11-20 19:17:04 -08:00
Frédéric Guillot c26787f476 Improve OPML package to be more idiomatic 2017-11-20 19:11:06 -08:00
Frédéric Guillot e91a9b4f13 Export only necessary structs in JsonFeed package 2017-11-20 18:57:54 -08:00
Frédéric Guillot 6618caca81 Use more idiomatic code for Atom parser 2017-11-20 18:50:16 -08:00
Frédéric Guillot 89307010ad Add parser for RDF feeds 2017-11-20 18:34:11 -08:00
Frédéric Guillot c5cd38de83 Add unit test for HTTP client response functions 2017-11-20 17:25:45 -08:00
Frédéric Guillot aecda64030 Make sure XML feeds are always encoded in UTF-8 2017-11-20 17:12:37 -08:00
Frédéric Guillot 0e6717b7c8 Ensure that LocalizedError are returned by parsers 2017-11-20 16:11:55 -08:00
Frédéric Guillot 557cf9c21d Handle RSS entries with Atom links 2017-11-20 15:48:26 -08:00
Frédéric Guillot cf8af56a99 Handle RSS feeds without entry links 2017-11-20 15:15:10 -08:00
Frédéric Guillot a76c2a8c22 Improve OPML import/export 2017-11-20 14:35:11 -08:00
Frédéric Guillot 8ffb773f43 First commit 2017-11-19 22:01:46 -08:00