Commit Graph

120 Commits

Author SHA1 Message Date
Fufu Fang 93f9701aa8 minor error message formatting 2019-04-26 13:32:59 +01:00
Fufu Fang 774f14c8e7 if we get HTTP 429, wait for 5 secs 2019-04-26 13:20:55 +01:00
Fufu Fang 9065dcffb5 minor bug fix 2019-04-26 11:43:35 +01:00
Fufu Fang edc3df733c minor bug fix 2019-04-26 11:42:11 +01:00
Fufu Fang 85d66adf6c added command line switch to configure user agent and the maximum number of network connections allowed 2019-04-26 11:39:03 +01:00
Fufu Fang f4fd419528 added command line option to change the number of the maximum number of network connection allowed 2019-04-26 11:27:39 +01:00
Fufu Fang 80a98a79f9 improved thread synchronisation, to stop Cache_read() from downloading the segment that is being downloaded in the background by Cache_bgdl() 2019-04-26 10:57:07 +01:00
Fufu Fang f479482d7b stop Cache_bgdl() from downloading past the last segment, remove excess debug message 2019-04-26 10:36:25 +01:00
Fufu Fang 5f6fc3f761 Now caches the directory structures by dumping LinkTables to the disk. 2019-04-26 08:26:47 +01:00
Fufu Fang 9a5f37d91f replaced strndupcat with path_append, added LinkTable_disk_*() functions 2019-04-26 07:39:45 +01:00
Fufu Fang 825bd4d3fb added the command line switch to control the download segment size 2019-04-25 21:34:07 +01:00
Fufu Fang b6b101bc63 removed unnecessary goto statements 2019-04-25 13:14:54 +01:00
Fufu Fang df025b1caf improved the correctness of Cache_bgdl() 2019-04-25 08:50:04 +01:00
Fufu Fang f5acebafe2 Merge branch 'master' of github.com:fangfufu/httpdirfs 2019-04-25 01:17:18 +01:00
Fufu Fang f628939713 Now dynamically allocate the receive buffers on the heap. 2019-04-25 00:58:29 +01:00
Fufu Fang 86999915e8
Merge pull request #29 from Lewiscowles1986/patch-1
Make larger block size
2019-04-24 23:59:02 +01:00
Fufu Fang e442871899 added Cache_background_download() to enable the download of the next block 2019-04-24 23:58:26 +01:00
Lewis Cowles 6d02b0554b
Make larger block size
The queue is inherently serial, not a queue data structure, and the code has no concurrency / parallelization added, but making the block-size larger should increase supported file-size & use disk cache more efficiently.
2019-04-24 06:22:43 +01:00
Fufu Fang 12c19e3421 improved Cache_read() 2019-04-24 04:28:54 +01:00
Fufu Fang 97ea20ef89 updated comment 2019-04-24 04:28:54 +01:00
Fufu Fang cc1697894b Resolved race condition - this code is now considered as stable.
Reference:
https://wiki.sei.cmu.edu/confluence/display/c/FIO24-C.+Do+not+open+a+file+that+is+already+open
2019-04-24 04:28:54 +01:00
Fufu Fang 2ebf37fb65 fixed doxygen warnings 2019-04-24 04:28:54 +01:00
Fufu Fang 797501514d changed the log output 2019-04-24 04:28:54 +01:00
Fufu Fang 60b2e6c0c8 minor cosmetic changes 2019-04-24 04:28:54 +01:00
Fufu Fang 0892cd0a8a changed header inclusion order 2019-04-24 04:28:54 +01:00
Fufu Fang 3a0c431bba Stability improvement
Metadata corruption still occasionally happens when it gets read in.
I suspect it is due to some sort of race condition.
2019-04-24 04:28:54 +01:00
Fufu Fang fb8a132c14 added a mutex lock for closing cache dataset 2019-04-24 04:28:54 +01:00
Fufu Fang cba66c58fd more stability and error handling improvements 2019-04-24 04:28:54 +01:00
Fufu Fang fe079f6266 improved stability / error handling 2019-04-24 04:28:54 +01:00
Fufu Fang 36ddbe9ec5 fixed bug in fs_open(), now handles cache dataset creation properly 2019-04-24 04:28:46 +01:00
Fufu Fang 6536163f98 improve cached dataset creation process, better error handling 2019-04-24 04:28:46 +01:00
Fufu Fang ac61ddd1c0 stability improvement, hopefully got rid off SIGFPE 2019-04-24 04:28:46 +01:00
Fufu Fang f122cf000d updated BUG section of README.md 2019-04-24 04:28:46 +01:00
Fufu Fang e166098162 updated README, also moved some status message around 2019-04-24 04:28:46 +01:00
Fufu Fang 96a1e4fd55 used a more stupid version of the bitmap, because it works reliably 2019-04-24 04:28:35 +01:00
Fufu Fang 0d8d5ef329 documented the new feature, and the bug 2019-04-24 04:28:35 +01:00
Fufu Fang e2d2b0dd28 Cache system has been succesfully implemented 2019-04-24 04:28:27 +01:00
Fufu Fang 692ad3f00c Cache dataset creation is working, time write the function to fill in the cache 2019-04-24 04:27:40 +01:00
Fufu Fang b6bdf15ad1 removed a bunch of function prototypes 2019-04-24 04:27:38 +01:00
Fufu Fang 77bb715590 fixed memory leak in link table creation 2019-04-24 04:21:38 +01:00
Fufu Fang 5af3def3f8 Linked in CacheSystem_init, Cache_open, Cache_close 2019-04-24 04:21:38 +01:00
Fufu Fang a02072f548 replaced fseek with fseeko, this should enable large file support 2019-04-24 04:21:38 +01:00
Fufu Fang 07b8382022 modified cache_create 2019-04-24 04:21:38 +01:00
Fufu Fang 212e319f20 unified struct member name between link.h and cache.h 2019-04-24 04:21:38 +01:00
Fufu Fang c6717e0450 changed a bunch of function to static function 2019-04-24 04:21:38 +01:00
Fufu Fang 86644c0518 added more documentation 2019-04-24 04:21:38 +01:00
Fufu Fang e9e41165ae added CacheDir_create() and fs_release() 2019-04-24 04:21:38 +01:00
Fufu Fang 0c55dc5a0c wrote Seg_exist, Seg_set, refined CacheSystem_init 2019-04-24 04:21:38 +01:00
Fufu Fang b1971a7c3f modified strndupcat, added more comments 2019-04-24 04:21:38 +01:00
Fufu Fang e5c77f3680 Changed the way segments work, added more functions
Now segments are fixed size block of 128KiB
2019-04-24 04:21:38 +01:00
Fufu Fang 3d0269e90f added Cache_create 2019-04-24 04:21:38 +01:00
Fufu Fang 4b60b4e9be wrote the cache opening function 2019-04-24 04:21:38 +01:00
Fufu Fang 794fb9414f wrote more Cache_open 2019-04-24 04:21:37 +01:00
Fufu Fang 7ae6d6fe1b all the functions written so far compile fine, working on Cache_open 2019-04-24 04:21:37 +01:00
Fufu Fang 9f1b963014 fixed documentation format 2019-04-24 04:21:37 +01:00
Fufu Fang 21f1cf2a18 changed strlen in link.c to strnlen, and changed LINK_LEN_MAX to P_URL_LEN_MAX 2019-04-24 04:21:37 +01:00
Fufu Fang d9cdc20b42 moved NETWORK_MAX_CONNS from network.h to network.c 2019-04-24 04:21:37 +01:00
Fufu Fang 7cafdbf7d4 Completed metadata read/write functions 2019-04-24 04:21:37 +01:00
Fufu Fang 8d10316b95 changed the data type for file size. wrote Data_read() 2019-04-24 04:21:37 +01:00
Fufu Fang c710904cfd wrote Data_create function 2019-04-24 04:21:37 +01:00
Fufu Fang 90eccb6049 created the skeletal cache.c 2019-04-24 04:21:37 +01:00
Fufu Fang 7b75e1ea41 enable http pipelining, changed max connection limit to 10 2019-04-24 04:21:37 +01:00
Fufu Fang c4c0f1fd9a i think i have finalised the cache.h 2019-04-24 04:21:37 +01:00
Fufu Fang f5361225cd removed limits.h
It is unnecessary, I am not sure why I included it in the first place.
2019-04-24 04:21:37 +01:00
Josh Lilly 8ea19295a0 Compatibility with FreeBSD 2019-04-24 04:21:37 +01:00
Fufu Fang e38bd0fa80 talked to fanf2 about the metadata in-memory data structure 2019-04-24 04:21:37 +01:00
Fufu Fang a3e2b0bc3b added some of the prototype for the cache controller 2019-04-24 04:21:37 +01:00
Fufu Fang b4554233dc fixed issue #24 2019-04-24 04:21:10 +01:00
Fufu Fang 4f582ae94f fixed issue #23 (Dotfile madness) 2019-04-24 04:21:10 +01:00
Jerome Charaoui f6f50c7cdf Move source files in src/ 2019-04-24 04:21:10 +01:00