Commit Graph

39 Commits

Author SHA1 Message Date
Lunny Xiao 9d99f6ab19
Refactor renders (#15175)
* Refactor renders

* Some performance optimization

* Fix comment

* Transform reader

* Fix csv test

* Fix test

* Fix tests

* Improve optimaziation

* Fix test

* Fix test

* Detect file encoding with reader

* Improve optimaziation

* reduce memory usage

* improve code

* fix build

* Fix test

* Fix for go1.15

* Fix render

* Fix comment

* Fix lint

* Fix test

* Don't use NormalEOF when unnecessary

* revert change on util.go

* Apply suggestions from code review

Co-authored-by: zeripath <art27@cantab.net>

* rename function

* Take NormalEOF back

Co-authored-by: zeripath <art27@cantab.net>
2021-04-19 18:25:08 -04:00
KN4CK3R c03e488e14
Add LFS Migration and Mirror (#14726)
* Implemented LFS client.

* Implemented scanning for pointer files.

* Implemented downloading of lfs files.

* Moved model-dependent code into services.

* Removed models dependency. Added TryReadPointerFromBuffer.

* Migrated code from service to module.

* Centralised storage creation.

* Removed dependency from models.

* Moved ContentStore into modules.

* Share structs between server and client.

* Moved method to services.

* Implemented lfs download on clone.

* Implemented LFS sync on clone and mirror update.

* Added form fields.

* Updated templates.

* Fixed condition.

* Use alternate endpoint.

* Added missing methods.

* Fixed typo and make linter happy.

* Detached pointer parser from gogit dependency.

* Fixed TestGetLFSRange test.

* Added context to support cancellation.

* Use ReadFull to probably read more data.

* Removed duplicated code from models.

* Moved scan implementation into pointer_scanner_nogogit.

* Changed method name.

* Added comments.

* Added more/specific log/error messages.

* Embedded lfs.Pointer into models.LFSMetaObject.

* Moved code from models to module.

* Moved code from models to module.

* Moved code from models to module.

* Reduced pointer usage.

* Embedded type.

* Use promoted fields.

* Fixed unexpected eof.

* Added unit tests.

* Implemented migration of local file paths.

* Show an error on invalid LFS endpoints.

* Hide settings if not used.

* Added LFS info to mirror struct.

* Fixed comment.

* Check LFS endpoint.

* Manage LFS settings from mirror page.

* Fixed selector.

* Adjusted selector.

* Added more tests.

* Added local filesystem migration test.

* Fixed typo.

* Reset settings.

* Added special windows path handling.

* Added unit test for HTTPClient.

* Added unit test for BasicTransferAdapter.

* Moved into util package.

* Test if LFS endpoint is allowed.

* Added support for git://

* Just use a static placeholder as the displayed url may be invalid.

* Reverted to original code.

* Added "Advanced Settings".

* Updated wording.

* Added discovery info link.

* Implemented suggestion.

* Fixed missing format parameter.

* Added Pointer.IsValid().

* Always remove model on error.

* Added suggestions.

* Use channel instead of array.

* Update routers/repo/migrate.go

* fmt

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: zeripath <art27@cantab.net>
2021-04-08 18:25:57 -04:00
zeripath a351b22dc0
Prevent NPE in CommentMustAsDiff if no hunk header (#15199)
I do not understand how this can happen or why.

There is an apparent possibility for a comment.Patch to be missing a hunk header
- this should not happen and do not understand how. But it appears to happen on
1.13 at least in some case.

This PR will simply add a new section if the cursection is empty
thus preventing the NPE.

Fix #15198

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2021-04-01 14:29:14 -04:00
KN4CK3R 0c6137617f
Add Tabular Diff for CSV files (#14661)
Implements request #14320 The rendering of CSV files does match the diff style.

* Moved CSV logic into base package.

* Added method to create a tabular diff.

* Added CSV compare context.

* Added CSV diff template.

* Use new table style in CSV markup.

* Added file size limit for CSV rendering.

* Display CSV parser errors in diff.

* Lazy read single file.

* Lazy read rows for full diff.

* Added unit tests for various CSV changes.
2021-03-29 22:44:28 +02:00
zeripath 46782d53bc
Place wrapper around comment as diff to catch panics (#15085)
There are a few recurrent issues with comment as diff reporting panics that are resistant to fixing due to the fact that the panic occurs in the template render and is swallowed by the template renderer.

This PR just adds some logging to force the panic to properly logged and re-propagates back up to the template renderer so we can actually detect what the issue is.

Signed-off-by: Andrew Thornton art27@cantab.net
2021-03-21 16:59:58 +00:00
zeripath f0e15250b9
Migrate to use jsoniter instead of encoding/json (#14841)
* Migrate to use jsoniter

* fix tests

* update gitea.com/go-chi/binding

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
2021-03-01 22:08:10 +01:00
zeripath 3d8b5ad5f3
Fix a couple of CommentAsPatch issues. (#14804)
* CutDiffAroundLine makes the incorrect assumption that `---` and `+++` always represent part of the header of a diff.

This PR adds a flag to its parsing to prevent this problem and adds a streaming parsing technique to CutDiffAroundLine using an io.pipe instead of just sending data to an unbounded buffer.

Fix #14711

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Handle unquoted comment patch files

When making comment patches unfortunately the patch does not always quote the filename
This makes the diff --git header ambiguous again.

This PR finally adds handling for ambiguity in to parse patch

Fix #14812

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add in testing for no error

There is no way currently for CutDiffAroundLine in this test to cause an
error however, it should still be tested.

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-27 19:46:14 +01:00
zeripath beb2058186
Fix broken spans in diffs (#14678)
Gitea runs diff on highlighted code fragment for each line in order to provide
code highlight diffs. Unfortunately this diff algorithm is not aware that span tags
and entities are atomic and cannot be split.

The current fixup code makes some attempt to fix these broken tags however, it cannot
handle situations where a tag is split over multiple blocks.

This PR provides a more algorithmic fixup mechanism whereby spans and entities are
completely coalesced into their respective blocks.

This may result in a incompletely reduced diff but - it will definitely prevent the
broken entities and spans that are currently possible.

As a result of this fixup several inconsistencies were discovered in our testcases
and these were also fixed.

Fix #14231

Signed-off-by: Andrew Thornton <art27@cantab.net>
2021-02-14 15:51:00 +01:00
vnkmpf 487f2ee41c
Whitespace in commits (#14650)
* Add whitespace to commit view

* Add whitespace to /compare/a...b

* Move repeated whitespaceFlags to gitdiff

* Add whitespace for wiki pages
2021-02-13 05:35:43 +01:00
zeripath 236e70f135
Fix escaping issue in diff (#14153)
Ensure that linecontent is escaped before passing to template.HTML

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-12-26 21:58:21 +00:00
zeripath 45ca2e4605
Handle incomplete diff files properly (#13662)
* Handle incomplete diff files properly

The code for parsing diff hunks has a bug whereby a very long line
in a very long diff would not be completely read leading to an unexpected
character.

This PR ensures that the line is completely cleared

Fix #13602

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Also allow git max line length <4096

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add test case

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-11-21 17:41:24 -05:00
mrsdizzie d7e0983780
Alternative fix for HTML diff entity split (#13425)
* Alternative fix for HTML diff entity split

This commit both reverts PR #13357 and uses the exiting implementation alredy used for spans to fix the same issue. That PR duplicates most of logic that is already present elsewhere and still was failing for some cases. This should be simpler as it uses the existing logic that already works for <span>s being split apart.

Added both test cases as well.

* Update gitdiff_test.go

* fmt

* entity can have uppercase letter, also add detailed comment per @zeripath
2020-11-04 19:35:05 -05:00
zeripath 6b7c199f5f
When creating line diffs do not split within an html entity (#13357)
* When creating line diffs do not split within an html entity

Fix #13342

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add test case

Signed-off-by: Andrew Thornton <art27@cantab.net>

* improve test

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-10-31 13:24:32 -04:00
zeripath 327f18c3b2
Fix initial commit page & binary munching problem (#13249)
* Fix initial commit page

Unfortunately as a result of properly fixing ParsePatch the hack that
used git show <initial_commit_id> to get the diff for this failed.

This PR fixes this using the "super-secret" empty tree ref to make the
diff against.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Also fix #13248

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update services/gitdiff/gitdiff.go

Co-authored-by: 6543 <6543@obermui.de>
2020-10-21 19:08:44 -04:00
mrsdizzie f6ee7ce9b6
Add better error checking for inline html diff code (#13239)
* Add better error checking for inline html diff code

A better fix for #13191 which cleans up this code a bit and adds basic checking which should avoid writing broken HTML in future situations.

* Update gitdiff_test.go

* better regex

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-10-21 18:14:44 -04:00
mrsdizzie 9fe4b7b692
Fix error in diff html rendering (#13191)
* Fix error in diff html rendering

Was missing an optional whitespace check in regex. Also noticed a rare case where diff.Type == Equal would be empty and thus get a newline attached. Fixed that too.

Fixes #13177

* Update services/gitdiff/gitdiff.go

Co-authored-by: zeripath <art27@cantab.net>

* Update gitdiff_test.go

* fmt

Co-authored-by: zeripath <art27@cantab.net>
2020-10-18 13:51:38 -04:00
zeripath 6bab678bed
Fix diff skipping lines (#13154)
* Fix diff skipping lines

ParsePatch previously just skipped all lines that start with "+++ " or "--- "
and makes no attempt to see these lines in context.

This PR rewrites ParsePatch to pay attention to context and position
within a patch, ensuring that --- and +++ are only skipped if
appropriate.

This PR also fixes several issues with incomplete files.

Fix https://codeberg.org/Codeberg/Community/issues/308
Fix #13153

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add testcase

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix comment

* simplify error handling

Signed-off-by: Andrew Thornton <art27@cantab.net>

* never return io.EOF

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
2020-10-16 13:13:18 -04:00
zeripath edfebe65b1
Finally fix diff names (#13136)
* Finally fix diff names

#12771 attempted to fix diff by avoiding the git diff line as
it is possible to have an ambiguous line here.

#12254 attempted to fix diff by assuming that names would quoted
if they needed to be and if one was quoted then both would be.

Both of these were wrong.

I have now discovered `--src-prefix` and `--dst-prefix` which
means that we can set this in such a way to force the git diff
to always be unambiguous.

Therefore this PR rollsback most of the changes in #12771 and
uses these options to fix this.

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Update services/gitdiff/gitdiff.go

* Update services/gitdiff/gitdiff.go

* Update modules/repofiles/temp_repo.go

* fix test

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
2020-10-14 07:49:33 +03:00
赵智超 dfa7291f8f
[Enhancement] Allow admin to merge pr with protected file changes (#12078)
* [Enhancement] Allow admin to merge pr with protected file changes

As tilte, show protected message in diff page and merge box.

Signed-off-by: a1012112796 <1012112796@qq.com>

* remove unused ver

* Update options/locale/locale_en-US.ini

Co-authored-by: Cirno the Strongest <1447794+CirnoT@users.noreply.github.com>

* Add TrN

* Apply suggestions from code review

* fix lint

* Update options/locale/locale_en-US.ini

Co-authored-by: zeripath <art27@cantab.net>

* Apply suggestions from code review

* move pr proteced files check to TestPatch
* Call TestPatch when protected branches settings changed

* Apply review suggestion @CirnoT

* move to service @lunny

* slightly restructure routers/private/hook.go

Adds a lot of comments and simplifies the logic

Signed-off-by: Andrew Thornton <art27@cantab.net>

* placate lint

Signed-off-by: Andrew Thornton <art27@cantab.net>

* skip duplicate protected files check

* fix check logic

* slight refactor of TestPatch

Signed-off-by: Andrew Thornton <art27@cantab.net>

* When checking for protected files changes in TestPatch use the temporary repository

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix introduced issue with hook

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Remove the check on PR index being greater than 0 as it unnecessary

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <matti@mdranta.net>
Co-authored-by: Cirno the Strongest <1447794+CirnoT@users.noreply.github.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-10-13 14:50:57 -04:00
zeripath 96969ddec8
Fix yet another bug with diff file names (#12771)
Following further testing it has become apparent that the diff line
cannot be used to determine filenames for diffs with any sort of predictability
the answer therefore is to use the other lines that are provided with a diff

Fix #12768

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-09-09 14:08:40 +01:00
zeripath 7a8a05cc44
Fix diff path unquoting (#12554)
* Fix diff path unquoting

services/gitdiff/gitdiff.go whereby there it assumed that the path would
always be quoted on both sides

This PR simplifies the code here and uses fmt.Fscanf to parse the
strings as necessary.

Fix #12546

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add testcase as per @mrsdizzie

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-08-22 16:07:37 +01:00
Gjergji Ramku 0c9eb468e9
Fix typos (#12542)
Signed-off-by: Gjergji Ramku <gjergjiramku@gmail.com>
2020-08-20 09:53:06 -05:00
mrsdizzie 73b155d5f2
Add more tests for diff highlighting (#12467)
Adding some more tests for different cases to make sure they keep working

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-08-11 00:29:25 -04:00
mrsdizzie 0171dda728
Improved diff syntax highlighting fix (#12455)
Make previous fix from #12238 more robust since I saw a case where a diff changes only a single character in a chroma class instead of the entire thing. Add another more complicated test to match.

Co-authored-by: Lauris BH <lauris@nix.lv>
2020-08-08 11:17:02 +01:00
zeripath 2f6aadffa8
Git 2.28 no longer permits diff with ... on unrelated branches (#12364)
* Git 2.28 no longer permits diff with ... on unrelated branches

Signed-off-by: Andrew Thornton <art27@cantab.net>

* need to check stderr
2020-07-29 13:53:04 -04:00
mrsdizzie 9542b7317d
Fix Syntax highlight for token change in added/deleted code (#12238)
* Fix Syntax highlight for token change in added/deleted code

For diffs we first syntax highlight the before and after line then use a 3rd party diff library to find the difference in them and create a substring based on that, which we then highlight a 2nd time to show the specific difference within a line that has changed. In a specific case if the diffrence also changes the chroma class it will split in the middle of the attr and cause broken HTML:

```
<span class="nx">oldtext<span>
<span class="k">var newtext<span>
```

Will then split on

```
<span class="
```

Where the difference starts, and produce something broken like:

```
<span class="<span class="removed-code">nx"oldtext</span></span

```

Fix that by detecting when it happens and putting the HTML back together properly before highlighting the added/deleted code.

Fixes #12235

* fix lint

* apply fix to all diff sections. Also handle case where insert/remove starts with a closing span

* Add a test for this new code

* remove comment

Co-authored-by: Lauris BH <lauris@nix.lv>
2020-07-16 16:58:54 +03:00
mrsdizzie 8d081950e6
Ensure syntax highlighting is the same inside diffs (#12205)
Make sure to end up with the same syntax highlighting inside various sections of diffs by processing the code first before detecting specific changes between the lines. Also try and make sure that when highlighting individual lines in a diff that it is tokenized the same as it would be when part of an entire file with more context.

Fixes: #12190
2020-07-11 13:43:12 +08:00
mrsdizzie a6168fa25d
Make copy/paste work for source code (#12191)
* Make copy/paste work for source code

Fix regression casued by #12047 so copy/paste works properly in all browsers.

Fixes #12184

Also while looking at this I saw a small display issue for blame view. I think #12023 was merged into original PR through an update branch before #12047 was merged and made one of the css ruules not apply anymore.

* use pseudo-element to prevent copying of comment + symbol even when not visually selected

* remove added newline here should not be necessary anymore

* make sure empty line is newline so there is something to select and copy
2020-07-08 22:02:38 +01:00
mrsdizzie af7ffaa279
Server-side syntax highlighting for all code (#12047)
* Server-side syntax hilighting for all code

This PR does a few things:

* Remove all traces of highlight.js
* Use chroma library to provide fast syntax hilighting directly on the server
* Provide syntax hilighting for diffs
* Re-style both unified and split diffs views
* Add custom syntax hilighting styling for both regular and arc-green

Fixes #7729
Fixes #10157
Fixes #11825
Fixes #7728
Fixes #3872
Fixes #3682

And perhaps gets closer to #9553

* fix line marker

* fix repo search

* Fix single line select

* properly load settings

* npm uninstall highlight.js

* review suggestion

* code review

* forgot to call function

* fix test

* Apply suggestions from code review

suggestions from @silverwind thanks

Co-authored-by: silverwind <me@silverwind.io>

* code review

* copy/paste error

* Use const for highlight size limit

* Update web_src/less/_repository.less

Co-authored-by: Lauris BH <lauris@nix.lv>

* update size limit to 1MB and other styling tweaks

* fix highlighting for certain diff sections

* fix test

* add worker back as suggested

Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
2020-07-01 00:34:03 +03:00
zeripath 6e4480835f
Fix issue with DiffIndex on initial commit (#11677)
Unfortunately #11614 introduced a bug whereby the initial commit of a
repository could not be seen due to there being no parent commit to
create a clear diff from.

Here we create a diffstat from the difference between the parentless SHA and the SHA of the empty tree - a constant known to git. (With thanks to @L0veSunshine for informing me of this SHA)

Thanks to @a1012112796 for initial attempt to fix.

Fix #11650

Closes #11674

Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-Authored-By: L0veSunshine <xuan199651@gmail.com>
2020-05-29 22:14:00 +01:00
zeripath 5cb201dc93
Fix numbr of files, total additions, and deletions (#11614)
Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-05-26 01:58:07 -04:00
Lunny Xiao e8860ef4f9 Some refactor on git diff and ignore getting commit information failed on migrating pull request review comments (#9996)
* Some refactor on git diff and ignore getting commit information failed on migrating pull request review comments

* fix test

* fix lint

* Change error log to warn
2020-01-28 08:02:03 +00:00
Lunny Xiao f6067a8465 Migrate reviews when migrating repository from github (#9463)
* fix typo

* Migrate reviews when migrating repository from github

* fix lint

* Added test and migration when external user login

* fix test

* fix commented state

* Some improvements

* fix bug when get pull request and ref original author on code comments

* Fix migrated line; Added comment for review

* Don't load all pull requests attributes

* Fix typo

* wrong change copy head

* fix tests

* fix reactions

* Fix test

* fix fmt

* fix review comment reactions
2020-01-23 19:28:15 +02:00
Lauris BH 3185a13ea5
Check if diff actually contains sections when rendering (#9926) 2020-01-22 09:20:58 +02:00
zeripath 60c5339042 Graceful: Cancel Process on monitor pages & HammerTime (#9213)
* Graceful: Create callbacks to with contexts

* Graceful: Say when Gitea is completely finished

* Graceful: Git and Process within HammerTime

Force all git commands to terminate at HammerTime
Force all process commands to terminate at HammerTime
Move almost all git processes to run as git Commands

* Graceful: Always Hammer after Shutdown

* ProcessManager: Add cancel functionality

* Fix tests

* Make sure that process.Manager.Kill() cancels

* Make threadsafe access to Processes and remove own unused Kill

* Remove cmd from the process manager as it is no longer used

* the default context is the correct context

* get rid of double till
2019-11-30 16:40:22 +02:00
Lunny Xiao ef9a0370ca
Fix panic when diff (#9187)
* fix panic when diff

* improve code
2019-11-28 17:05:57 +08:00
Benno 149a9df9e8 Expand/Collapse Files and Blob Excerpt while Reviewing/Comparing code (#8924)
* update #8659 fold/unfold code diffs

* add fold button style

* update #8659 implement expand up/down codes (blob excerpt)

* fix golint errors

* fix expand direction

* remove debug message

* update css style for blob exceprt

* fix typo in comment

* update style sheet with less

* update expect diff (add SectionInfo)

* update #8942 accept suggested change (fix typo)

* close reader and check file type before get tail section

* adjust button position and check file type before insert fold button

* move index js to web_src

* merge index.js with master

* generate index.js

* update js coding style
2019-11-15 10:52:59 +08:00
zeripath 722a7c902d
Add Close() method to gogitRepository (#8901)
In investigating #7947 it has become clear that the storage component of go-git repositories needs closing.

This PR adds this Close function and adds the Close functions as necessary.

In TransferOwnership the ctx.Repo.GitRepo is closed if it is open to help prevent the risk of multiple open files.

Fixes #7947
2019-11-13 07:01:19 +00:00
Lunny Xiao c03d75fbd5 Move git diff codes from models to services/gitdiff (#7889)
* move git diff codes from models to services/gitdiff

* fix template

* fix test

* fix template
2019-09-05 22:20:09 -04:00