Frédéric Guillot
2f3708d40c
Do not use charset.NewReader if the body is a valid UTF-8 document
2020-10-30 23:00:05 -07:00
Lee Tang
46c13b5185
Restore the ability to use a proxy for all HTTP requests
...
The default transport was changed in commit 0d66f2chttps://golang.org/src/net/http/transport.go#L43
- https://golang.org/pkg/net/http/#ProxyFromEnvironment
2020-10-30 19:03:41 -07:00
Frédéric Guillot
16b7b3bc3e
http client: remove dependency on global config options
2020-09-27 14:37:46 -07:00
Frédéric Guillot
0d66f2c6d3
Tweak default HTTP client transport timeout values
...
Reducing these values avoid going over the max number of file descriptors when refreshing lot of feeds
2020-09-27 13:20:48 -07:00
Kebin Liu
cf7712acea
Add HTTP proxy option for subscriptions
2020-09-09 23:28:54 -07:00
Frédéric Guillot
6c6ca69141
Add feed option to ignore HTTP cache
2020-06-05 22:04:52 -07:00
Frédéric Guillot
3debf75eb9
Normalize URL query string before executing HTTP requests
...
- Make sure query strings parameters are encoded
- As opposed to the standard library, do not append equal sign
for query parameters with empty value
- Strip URL fragments like Web browsers
2019-12-26 15:56:59 -08:00
Frédéric Guillot
ca48f7612a
Ignore invalid content type
2019-09-18 22:32:29 -07:00
Frédéric Guillot
bb720c87c1
Make HTTP Client timeout and max body size configurable
2019-06-02 07:29:56 -07:00
Peter De Wachter
15505ee4a2
Make UTF-8 the default encoding for XML feeds
...
Consider the feed http://planet.haskell.org/atom.xml
- This is a UTF-8 encoded XML file
- No encoding declaration in the XML header
- No Unicode byte order mark
- Served with HTTP Content-Type "text/xml" (no charset parameter)
Miniflux lets charset.NewReader handle this. The charset package
implements the HTML5 character encoding algorithm, which, in this
situation, defaults to windows-1252 encoding if there are no UTF-8
characters in the first 1000 bytes. So for this feed, we get the wrong
encoding.
I inserted an explicit "utf8.Valid()" check, which fixes this problem.
2019-01-02 21:05:05 -08:00
Frédéric Guillot
6ae935309a
Ignore JSON feeds from EnsureUnicode()
2018-12-12 21:37:39 -08:00
Frédéric Guillot
82e08d0f69
Update XML encoding regex to take single quotes into consideration
2018-12-12 21:13:06 -08:00
Frédéric Guillot
f3bff76aa1
Make sure slice is not out of range when reading XML prolog
2018-11-24 12:17:00 -08:00
Frédéric Guillot
ae1dc1a91e
Handle more encoding conversion edge cases
2018-10-29 23:00:03 -07:00
Frédéric Guillot
5870f04260
Simplify feed parser and format detection
...
- Avoid doing multiple buffer copies
- Move parser and format detection logic to its own package
2018-10-14 11:46:41 -07:00
Frédéric Guillot
9dc38a0803
Add missing package descriptions for GoDoc
2018-10-08 17:32:17 -07:00
Patrick
2538eea177
Add the possibility to override default user agent for each feed
2018-09-19 18:19:24 -07:00
Frédéric Guillot
dbcc5d8a97
Use canonical imports
2018-08-24 21:56:39 -07:00
Frédéric Guillot
17054b396e
Update user agent with new website URL
2018-07-28 10:32:48 -07:00
Frédéric Guillot
9c0f882ba0
Add specific 404 and 401 error messages
2018-06-30 12:42:12 -07:00
Frédéric Guillot
bddca15b69
Add new fields for feed username/password
2018-06-19 22:58:29 -07:00
Frédéric Guillot
b5c317f971
Disable keep-alive for HTTP client
2018-06-19 20:21:24 -07:00
Frédéric Guillot
7914261437
Close HTTP response body even for failed requests
2018-06-19 20:13:13 -07:00
Frédéric Guillot
2f4cd59ad9
Make sure to close request body in HTTP client
2018-04-29 23:11:10 -07:00
Frédéric Guillot
1eba1730d1
Move HTTP client to its own package
2018-04-28 10:51:07 -07:00
