rdelaage
/
miniflux-v2
mirror of https://github.com/miniflux/v2.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
672 Commits 1 Branch 65 Tags 66 MiB
Commit Graph

153 Commits

Author SHA1 Message Date
Frédéric Guillot 56efd2eb3f Add workaround for non GMT dates (RFC822, RFC850, and RFC1123) 
RFC822, RFC850, and RFC1123 are supposed to be always in GMT.

This is a workaround for the one defined in PST timezone.
 2018-12-26 20:24:38 -08:00
Frédéric Guillot 012138179c Add function storage.UpdateFeedError() 2018-12-15 13:04:38 -08:00
Tom Matthews 8b40778ee1 Add BBC News scraping rule 2018-12-13 20:25:30 -08:00
Frederic Guillot 61bfb3cfa8 Make password prompt compatible with Windows 2018-12-09 17:44:33 -08:00
Frédéric Guillot 1bc8535dbb Move image proxy filter to template functions 2018-12-02 21:09:53 -08:00
Frédéric Guillot 6f5d93cbbe Update scraper rule for lemonde.fr 2018-12-02 20:53:22 -08:00
Frédéric Guillot 311a133ab8 Refactor manual entry scraper 2018-12-02 20:51:06 -08:00
mapl e47188eab2 Update scraper rule for heise.de 2018-12-01 11:49:30 -08:00
Frédéric Guillot 487852f07e Replace daemon and scheduler package with service package 2018-11-11 15:32:48 -08:00
Frédéric Guillot 3b6e44c331 Allow the scraper to parse XHTML documents 
Only "text/html" was authorized before.
 2018-11-03 13:44:13 -07:00
Frédéric Guillot ae1dc1a91e Handle more encoding conversion edge cases 2018-10-29 23:00:03 -07:00
Frédéric Guillot 7d1b471d88 Add test case to check different feed encoding and HTTP headers 2018-10-29 19:04:36 -07:00
Frédéric Guillot 85d48c8a71 Add entries storage error to feed errors count 2018-10-21 11:44:29 -07:00
Frédéric Guillot b8f874a37d Simplify feed entries filtering 
- Rename processor package to filter
- Remove boilerplate code
 2018-10-14 22:33:19 -07:00
Frédéric Guillot 778346b0b0 Simplify feed fetcher 
- Add browser package to handle HTTP errors
- Reduce code duplication
 2018-10-14 21:43:48 -07:00
Frédéric Guillot 5870f04260 Simplify feed parser and format detection 
- Avoid doing multiple buffer copies
- Move parser and format detection logic to its own package
 2018-10-14 11:46:41 -07:00
Frédéric Guillot 9606126196 Convert text links and line feeds to HTML in YouTube channels 2018-10-08 20:47:10 -07:00
Frédéric Guillot 9dc38a0803 Add missing package descriptions for GoDoc 2018-10-08 17:32:17 -07:00
Frédéric Guillot 11dfcdd3d6 Fix typo in license header 2018-10-08 15:50:15 -07:00
Frédéric Guillot b1e8f534ef Simplify locale package usage (refactoring) 2018-09-22 15:04:55 -07:00
Frédéric Guillot beb7a0cfcb Use unique translation IDs instead of English text as key 2018-09-21 22:23:23 -07:00
Patrick 2538eea177 Add the possibility to override default user agent for each feed 2018-09-19 18:19:24 -07:00
Frédéric Guillot df2bebaf3d Update scraper rule for heise.de 2018-08-25 10:33:18 -07:00
Frédéric Guillot dbcc5d8a97 Use canonical imports 2018-08-24 21:56:39 -07:00
neepl 5365f31e90 Add support for published tag in Atom feeds 2018-07-17 21:52:05 -07:00
Frédéric Guillot a786e78aca Add embedly.com to iframe whitelist 2018-07-10 20:56:54 -07:00
dzaikos 6d25e02cb5 New `add_dynamic_image` rewriter for JavaScript-loaded images. 
Searches tags for various `data-*` attributes and sets `img` tag `src` attribute appropriately. Falls back to searching `noscript` for `img` tags.

Includes unit tests.
 2018-07-09 01:22:48 -04:00
dzaikos e1c56b2e53 Processor: Do rewriter before sanitizer for `entry.Content`. 
Addresses #163.
 2018-07-06 00:17:07 -04:00
Frédéric Guillot de1a4aad30 Add support for protocol relative YouTube URLs 2018-07-04 22:45:44 -07:00
dzaikos 7d4a195519 Sandbox iframes when sanitizing. 
Updated iframe unit tests.

Refactored sanitizer.getExtraAttributes() to use `switch` instead of multiple `if` statements.
 2018-07-03 12:55:18 -07:00
Frédéric Guillot 9c0f882ba0 Add specific 404 and 401 error messages 2018-06-30 12:42:12 -07:00
dzaikos 45d7105ed1 Refactor AddImageTitle rewriter. 
* Only processes images with `src` **and** `title` attributes (others are ignored).
* Processes **all** images in the document (not just the first one).
* Wraps the image and its title attribute in a `figure` tag with the title attribute's contents in a `figcaption` tag.

Updated xkcd rewriter unit test.

Added another xkcd rewriter unit test to check rendering of images without title tags.
 2018-06-26 17:50:18 -04:00
dzaikos c9131b0e89 Improve sanitizer to remove style tag contents. 
See #157.

Refactored how blacklisted tags are handled so they're easier manage in the future.
 2018-06-24 19:53:23 -07:00
Dave Z d847b10e32 Improve sanitizer to remove script and noscript contents 
These tags where removed but the content was rendered as escaped HTML.

See #157
 2018-06-23 17:50:43 -07:00
Frédéric Guillot bddca15b69 Add new fields for feed username/password 2018-06-19 22:58:29 -07:00
Frédéric Guillot c719cf7df0 Rewrite iframe Youtube URLs to https://www.youtube-nocookie.com 2018-06-12 18:45:09 -07:00
Frédéric Guillot 0c2e5ff0dc Handle feeds with dates formatted as Unix timestamp 2018-05-08 20:41:24 -07:00
Frédéric Guillot 5cacae6cf2 Add API endpoint to import OPML file 2018-04-29 18:56:40 -07:00
Frédéric Guillot 1eba1730d1 Move HTTP client to its own package 2018-04-28 10:51:07 -07:00
aniran 322b265d7a Scrape parent element for iframe 
Current behavior: if you have an `iframe` scraper rule, `scrapContent`
tries to return the inner HTML of the `iframe`, which turns up blank.

New behavior: like `img` elements, if an `iframe` is matched by a scraper rule,
the parent element's inner HTML (i.e. the `iframe` is returned).
 2018-04-27 17:57:22 -07:00
aniran 920dda79b7 Add soundcloud and bandcamp iframe sources 2018-04-27 17:55:58 -07:00
Frédéric Guillot dcbb5047b1 Add support for Dublin Core date in RDF feeds 2018-04-10 18:13:05 -07:00
Frédéric Guillot 02ba735ba9 Handle some non-english date formats 2018-04-09 21:27:15 -07:00
Frédéric Guillot e2d02bac5a Rename RSS parser getters 2018-04-09 20:38:12 -07:00
Frédéric Guillot f76093690c Get the right comments URL when having multiple namespaces 2018-04-09 20:30:55 -07:00
Frédéric Guillot 702256bcc0 Add unit test for comments url and French translation 2018-04-07 13:56:11 -07:00
Ben Brooks 538d08c16c Add CommentsURL to entry 2018-04-07 13:50:45 -07:00
Frédéric Guillot 6ea4da3bce Handle RSS author elements with inner HTML 2018-03-18 11:57:46 -07:00
Frédéric Guillot 482785c5e6 Convert enclosure size field to bigint 2018-03-14 20:09:06 -07:00
Frédéric Guillot ec08f45bf5 Fix broken OPML import with Go 1.10 2018-03-14 18:50:06 -07:00
Frédéric Guillot f110384f11 Improve parser error messages 2018-02-27 21:19:59 -08:00
Frédéric Guillot 953d0a2dc0 Support localized feed errors generated by background workers 2018-02-27 21:08:32 -08:00
Frédéric Guillot 9292d5d604 Handle Atom feeds with HTML title 2018-02-17 12:21:58 -08:00
Frédéric Guillot dda9114692 Improve error handling for HTTP client 2018-02-08 18:16:54 -08:00
Frédéric Guillot 7b0bfd9308 Strip invalid XML characters to avoid parsing errors 2018-02-07 20:57:56 -08:00
Frédéric Guillot c6fd9eb9b1 Remove period for feed errors 2018-02-07 19:10:36 -08:00
Frédéric Guillot 0fb87eba3f Improve error handling when the response is empty 2018-02-07 18:47:47 -08:00
Frédéric Guillot b78172033f Show API URL endpoints in user interface 2018-01-31 21:57:20 -08:00
Frédéric Guillot ffabb009b8 Do not override existing entries when the crawler is enabled 2018-01-20 14:04:19 -08:00
Frédéric Guillot 713b38e34c Handle more encoding edge cases 
- Feeds with charset specified only in Content-Type header and not in XML document
- Feeds with charset specified in both places
- Feeds with charset specified only in XML document and not in HTTP header
 2018-01-20 13:25:21 -08:00
Frédéric Guillot 3b62f904d6 Do not crawl existing entry URLs 2018-01-20 13:25:20 -08:00
Frédéric Guillot 9652dfa1fe Add more comments (GoDoc) 2018-01-11 19:21:20 -08:00
Frédéric Guillot 1d7fe892e1 Add scraper rule for darkreading.com 2018-01-06 13:25:12 -08:00
Frédéric Guillot 48aa0d07ef Add more scraper rules 2018-01-04 19:32:24 -08:00
Frédéric Guillot 7d278d49f1 Add content length check when refreshing feeds 2018-01-04 18:41:23 -08:00
Frédéric Guillot efac11e082 Handle more date formats 2018-01-03 18:59:29 -08:00
Frédéric Guillot ec63cbe7bb If the website URL is empty, assign the feed URL 2018-01-03 18:23:21 -08:00
Frédéric Guillot c39f2e1a8d Rename helper packages 2018-01-02 19:15:08 -08:00
Frédéric Guillot 3c3f397bf5 Make sure the scraper parse only HTML documents 2018-01-02 18:32:01 -08:00
Frédéric Guillot c454f67037 Add scraper rules for version2.dk and ing.dk 2017-12-27 19:44:23 -08:00
Frédéric Guillot d4839b5597 Add more scraper rules 2017-12-27 13:36:07 -08:00
Frédéric Guillot f6a5d7d6ed Add support for data URL favicons 2017-12-22 19:01:39 -08:00
Frédéric Guillot e7afec7eca Handle more date formats 2017-12-22 17:59:28 -08:00
Frédéric Guillot 1d8193b892 Add logger 2017-12-15 18:55:57 -08:00
Frédéric Guillot c6d9eb3614 Improve content scraper 2017-12-13 21:30:40 -08:00
Frédéric Guillot 827683ab59 Make sure that item URL are absolute 2017-12-13 20:16:15 -08:00
Frédéric Guillot 84d912c979 Rewrite imports 2017-12-12 21:48:13 -08:00
Frédéric Guillot ef097f02fe Add the possibility to enable crawler for feeds 2017-12-12 19:19:36 -08:00
Frédéric Guillot 33445e5b68 Add the possibility to define rewrite rules for each feed 2017-12-11 22:16:32 -08:00
Frédéric Guillot 87ccad5c7f Add scraper rules 2017-12-10 20:51:04 -08:00
Frédéric Guillot 7a35c58f53 Add readability package to fetch original content 2017-12-10 19:01:38 -08:00
Frédéric Guillot 6f5350a497 Move packages http and url 2017-12-02 20:26:21 -08:00
Frédéric Guillot 2356ddad28 Add Pinboard integration 2017-12-02 19:32:14 -08:00
Frédéric Guillot fb2a73c91e Proxify image enclosures 2017-12-01 22:29:18 -08:00
Frédéric Guillot bb8e61c7c5 Make sure golint pass on the code base 2017-11-27 21:40:05 -08:00
Frédéric Guillot bd663b43a0 Improve HTML sanitizer 2017-11-25 18:08:59 -08:00
Frédéric Guillot 71bf7e4358 Improve API 2017-11-24 22:29:20 -08:00
Frédéric Guillot 2b641cc224 Improve feed parsers 2017-11-22 14:52:31 -08:00
Frédéric Guillot 99dfbdbb47 Convert feed encoding only if the charset is specified 2017-11-21 22:55:19 -08:00
Frédéric Guillot 5f0ae8196c Add timeout for HTTP client 2017-11-20 19:44:28 -08:00
Frédéric Guillot eb9f588216 Make sure RDF entries have a date 2017-11-20 19:25:30 -08:00
Frédéric Guillot d5838b6734 Move feed parsers packages in reader package 2017-11-20 19:17:04 -08:00
Frédéric Guillot c26787f476 Improve OPML package to be more idiomatic 2017-11-20 19:11:06 -08:00
Frédéric Guillot e91a9b4f13 Export only necessary structs in JsonFeed package 2017-11-20 18:57:54 -08:00
Frédéric Guillot 6618caca81 Use more idiomatic code for Atom parser 2017-11-20 18:50:16 -08:00
Frédéric Guillot 89307010ad Add parser for RDF feeds 2017-11-20 18:34:11 -08:00
Frédéric Guillot c5cd38de83 Add unit test for HTTP client response functions 2017-11-20 17:25:45 -08:00
Frédéric Guillot aecda64030 Make sure XML feeds are always encoded in UTF-8 2017-11-20 17:12:37 -08:00
Frédéric Guillot 0e6717b7c8 Ensure that LocalizedError are returned by parsers 2017-11-20 16:11:55 -08:00
Frédéric Guillot 557cf9c21d Handle RSS entries with Atom links 2017-11-20 15:48:26 -08:00