Git will and can pack references into packfiles and therefore if you write/read the
files directly you will get false results. Instead you should use update-ref and
show-ref. To that end I have created three new functions in git/repo_commit.go that
will do this correctly.
Related #17191
Signed-off-by: Andrew Thornton <art27@cantab.net>
There are repeated panics in tests due to TestRepository_GetTag failing
to run properly. This happens when we attempt to reset the internal
repo for a tag which has failed to load. The problem is - the panic that
this is causing is preventing us from finding what the real error is.
This PR simply moves the failure out so we have a chance to see what
really is failing.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Prevent off-by-one error on comments on newly appended lines
There was a bug in CutDiffAroundLine whereby if a file without a terminal new line
has a patch which appends lines to it and a comment is placed on one of those lines
the comment diff will be a line out of place.
This fixes CutDiffAroundLine to simply ignore the missing terminal newline - however,
we should really improve this rendering to add a marker to say that there was a
previously missing terminal newline.
Fix#17875
Signed-off-by: Andrew Thornton <art27@cantab.net>
The current TestPatch conflict code uses a plain git apply which does not properly
account for 3-way merging. However, we can improve things using `git read-tree -m` to
do a three-way merge then follow the algorithm used in merge-one-file. We can also use
`--patience` and/or `--histogram` to generate a nicer diff for applying patches too.
Fix#13679Fix#6417
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR contains multiple fixes. The most important of which is:
* Prevent hang in git cat-file if the repository is not a valid repository
Unfortunately it appears that if git cat-file is run in an invalid
repository it will hang until stdin is closed. This will result in
deadlocked /pulls pages and dangling git cat-file calls if a broken
repository is tried to be reviewed or pulls exists for a broken
repository.
Fix#14734Fix#9271Fix#16113
Otherwise there are a few small other fixes included which this PR was initially intending to fix:
* Fix panic on partial compares due to missing PullRequestWorkInProgressPrefixes
* Fix links on pulls pages due to regression from #17551 - by making most /issues routes match /pulls too - Fix#17983
* Fix links on feeds pages due to another regression from #17551 but also fix issue with syncing tags - Fix#17943
* Add missing locale entries for oauth group claims
* Prevent NPEs if ColorFormat is called on nil users, repos or teams.
The current implementation of checkBranchName is highly inefficient
involving opening the repository, the listing all of the branch names
checking them individually before then using using opened repo to get
the tags.
This PR avoids this by simply walking the references from show-ref
instead of opening the repository (in the nogogit case).
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR registers requests with the process manager and manages hierarchy within the processes.
Git repos are then associated with a context, (usually the request's context) - with sub commands using this context as their base context.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Use check attribute code to check the assigned language of a file and send that in to
chroma as a hint for the language of the file.
Signed-off-by: Andrew Thornton <art27@cantab.net>
There are multiple places where Gitea does not properly escape URLs that it is building and there are multiple places where it builds urls when there is already a simpler function available to use this.
This is an extensive PR attempting to fix these issues.
1. The first commit in this PR looks through all href, src and links in the Gitea codebase and has attempted to catch all the places where there is potentially incomplete escaping.
2. Whilst doing this we will prefer to use functions that create URLs over recreating them by hand.
3. All uses of strings should be directly escaped - even if they are not currently expected to contain escaping characters. The main benefit to doing this will be that we can consider relaxing the constraints on user names and reponames in future.
4. The next commit looks at escaping in the wiki and re-considers the urls that are used there. Using the improved escaping here wiki files containing '/'. (This implementation will currently still place all of the wiki files the root directory of the repo but this would not be difficult to change.)
5. The title generation in feeds is now properly escaped.
6. EscapePound is no longer needed - urls should be PathEscaped / QueryEscaped as necessary but then re-escaped with Escape when creating html with locales Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
There is a slight race in checking of a context deadline exceed in #16467
which leads to a 500 on the repository page.
The solution is to check the error coming back from `*LogNameStatusRepoParser.Next()`
and if it is the `ContextDeadlineExceeded` break from the loop.
Fix#17314
Signed-off-by: Andrew Thornton <art27@cantab.net>
core.protectNTFS protects NTFS from files which may be difficult to remove or interact
with using the win32 api, however, it also appears to prevent such files from
being entered into the git indexes - fundamentally causing breakages with PRs that
affect these files. However, deliberately setting this to false may cause security
issues due to the remain sparse checkout of files in the merge pipeline.
The only sensible option therefore is to provide an optional setting which admins
could set which would forcibly switch this off if they are affected by this issue.
Fix#17092
Signed-off-by: Andrew Thornton <art27@cantab.net>
- Update default branch if needed
- Update protected branch if needed
- Update all not merged pull request base branch name
- Rename git branch
- Record this rename work and auto redirect for old branch on ui
Signed-off-by: a1012112796 <1012112796@qq.com>
Co-authored-by: delvh <dev.lh@web.de>
One of the biggest reasons for slow repository browsing is that we wait
until last commit information has been generated for all files in the
repository.
This PR proposes deferring this generation to a new POST endpoint that
does the look up outside of the main page request.
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR changes the compare page to make the "..." in the between branches a clickable
link. This changes the comparison type from "..." to "..". Similarly it makes the
initial compare icon clickable to switch the head and base branches.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Ignore Sync errors on pipes when doing `CheckAttributeReader.CheckPath`
* apply env patch
* Drop the Sync and fix a number of issues with the Close function
Signed-off-by: Andrew Thornton <art27@cantab.net>
* add logs for DBIndexer and CheckPath
* Fix some more closing bugs
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add test case for language_stats
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update modules/indexer/stats/db.go
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
Some people still appear to report unclosed cat-files. This PR simply adds the caller
to the process descriptor for the CatFileBatch and CatFileBatchCheck calls.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Fixes#16381
Note that changes to unprotected files via the web editor still cannot be pushed directly to the protected branch. I could easily add such support for edits and deletes if needed. But for adding, uploading or renaming unprotected files, it is not trivial.
* Extract & Move GetAffectedFiles to modules/git
When the external context is cancelled it is possible for the
GitLogReader to not itself be Closed.
This PR does three things:
1. Instead of adding a plain defer it wraps the `g.Close` in a func as
`g` may change.
2. It adds the missing explicit g.Close - although the defer fix makes
this unnecessary.
3. It passes down the external context as the base context for the
GitLogReader meaning that the cancellation of the external context will
pass down automatically.
Fix#17007
Signed-off-by: Andrew Thornton <art27@cantab.net>
Replaces #16262
Replaces #16250
Replaces #14833
This PR first implements a `git check-attr` pipe reader - using `git check-attr --stdin -z --cached` - taking account of the change in the output format in git 1.8.5 and creates a helper function to read a tree into a temporary index file for that pipe reader.
It then wires this in to the language stats helper and into the git diff generation.
Files which are marked generated will be folded by default.
Fixes#14786Fixes#12653
* make sure headGitRepo is closed on err too
* refactor
* Fix git.Blob.DataAsync(): exec cancel since we already read all bytes (close pipe since we return a NopCloser)
* Add proxy settings and support for migration and webhook
* Fix default value
* Add newline for example ini
* Add lfs proxy support
* Fix lint
* Follow @zeripath's review
* Fix git clone
* Fix test
* missgin http requests for proxy
* use empty
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
* Add info about list endpoints to CONTRIBUTING.md
* Let all list endpoints return X-Total-Count header
* Add TODOs for GetCombinedCommitStatusByRef
* Fix models/issue_stopwatch.go
* Rrefactor models.ListDeployKeys
* Introduce helper func and use them for SetLinkHeader related func
* Fix modified files list in webhooks when there is a space
There is an unfortunate bug with GetCommitFileStatus where files with
spaces are misparsed and split at the space.
There is a second bug because modern gits detect renames meaning that
this function no longer works correctly.
There is a third bug in that merge commits don't have their modified
files detected correctly.
Fix#15865
Signed-off-by: Andrew Thornton <art27@cantab.net>
Following the merging of https://github.com/go-git/go-git/pull/330 we
can now add a setting to avoid go-git reading and caching large objects.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Use `..` instead of `...` with `rev-list`. In combination with #16282 the receiver can get the correct commit. The behaviour is now like Github.
fixes#11802
There is a bug with last commit cache recursive cache where the last
commit information that refers to the current tree itself will cause a
panic due to its path ("") not being included in the expected tree entry
paths.
This PR fixes this by skipping the missing entry.
Fix#16290
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
remove log() func from gogs times and switch to proper logging
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Andrew Thornton <art27@cantab.net>
* Fix 500 Error with branch and tag sharing the same name #15592
Fixed 500 error while create Pull request when there are more
than one sources (branch, tag) with the same name
Fix#15592
Signed-off-by: Viktor Yakovchuk <viktor@yakovchuk.net>
* fix logging
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: 6543 <6543@obermui.de>
* More efficiently parse shas for shaPostProcessor
The shaPostProcessor currently repeatedly calls git rev-parse --verify on both backends
which is fine if there is only one thing that matches a sha - however if there are
multiple things then this becomes wildly inefficient.
This PR provides functions for both backends which are much faster to use.
Fix#16092
* Add ShaExistCache to RenderContext
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
* Improve get last commit using git log --name-status
git log --name-status -c provides information about the diff between a
commit and its parents. Using this and adjusting the algorithm to use
the first change to a path allows for a much faster generation of commit
info.
There is a subtle change in the results generated but this will cause
the results to more closely match those from elsewhere.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
* Make modules/context.Context a context.Context
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Simplify context calls
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Set the base context for requests to the HammerContext
Signed-off-by: Andrew Thornton <art27@cantab.net>
* pass context into get-last-commit
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Make commit_info cancellable
Signed-off-by: Andrew Thornton <art27@cantab.net>
* use context as context
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
* Added type sniffer.
* Switched content detection from base to typesniffer.
* Added GuessContentType to Blob.
* Moved image info logic to client.
Added support for SVG images in diff.
* Restore old blocked svg behaviour.
* Added missing image formats.
* Execute image diff only when container is visible.
* add margin to spinner
* improve BIN tag on image diffs
* Default to render view.
* Show image diff on incomplete diff.
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
Unfortunately some old repositories can have tags with empty Tagger, Commit
or Author. Go-Git variants will always have empty values for these whereas
the native git variant leaves them at nil. The simplest solution is just to
always have these set to empty Signatures.
v156 migration also makes the incorrect assumption that these cannot be empty.
Therefore add some handling to this and add logging and adjust broken
logging elsewhere in this migration.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Improve performance when there are multiple commits in the last commit cache
* read refs directly if we can
Signed-off-by: Andrew Thornton <art27@cantab.net>
Fix#15236
* Do not do 40byte conversion within ParseTreeLine
* Missed a to40ByteSHA
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Andrew Thornton <art27@cantab.net>
There is a weird gotcha with GetTagCommitID that because it uses git rev-list
can cause an ambiguous argument error.
This PR simply makes tags use the same code as branches.
Signed-off-by: Andrew Thornton <art27@cantab.net>
`enry.IsVendor` is kinda slow as it simply iterates across all regexps.
This PR ajdusts the regexps to combine them to make this process a
little quicker.
Related #15143
Signed-off-by: Andrew Thornton <art27@cantab.net>
* response 404 for diff/patch of a commit that not exist
fix#15217
Signed-off-by: a1012112796 <1012112796@qq.com>
* Update routers/repo/commit.go
Co-authored-by: silverwind <me@silverwind.io>
* use ctx.NotFound()
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
* Make auto check manual merge as a chooseable mod and add manual merge way on ui
as title, Before this pr, we use same way with GH to check manually merge.
It good, but in some special cases, misjudgments can occur. and it's hard
to fix this bug. So I add option to allow repo manager block "auto check manual merge"
function, Then it will have same style like gitlab(allow empty pr). and to compensate for
not being able to detect THE PR merge automatically, I added a manual approach.
Signed-off-by: a1012112796 <1012112796@qq.com>
* make swager
* api support
* ping ci
* fix TestPullCreate_EmptyChangesWithCommits
* Apply suggestions from code review
Co-authored-by: zeripath <art27@cantab.net>
* Apply review suggestions and add test
* Apply suggestions from code review
Co-authored-by: zeripath <art27@cantab.net>
* fix build
* test error message
* make fmt
* Fix indentation issues identified by @silverwind
Co-authored-by: silverwind <me@silverwind.io>
* Fix tests and make manually merged disabled error on API the same
Signed-off-by: Andrew Thornton <art27@cantab.net>
* a small nit
* fix wrong commit id error
* fix bug
* simple test
* fix test
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Extract out the common cat-file batch calls
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move bleve and elastic indexers to use a common cat-file --batch when indexing
Signed-off-by: Andrew Thornton <art27@cantab.net>
* move catfilebatch to batch_reader and rename to batch_reader.go
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
Unfortunately `git log revision ... --skip=x -- path` skips the number of commits
not the number of commits relating to the path.
This PR changes the function to have a reader that reads and skips the
necessary number of commits by hand instead.
Fix#8716
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Support create single tag directly
support create tag with message from create release ui
Signed-off-by: a1012112796 <1012112796@qq.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
* CutDiffAroundLine makes the incorrect assumption that `---` and `+++` always represent part of the header of a diff.
This PR adds a flag to its parsing to prevent this problem and adds a streaming parsing technique to CutDiffAroundLine using an io.pipe instead of just sending data to an unbounded buffer.
Fix#14711
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Handle unquoted comment patch files
When making comment patches unfortunately the patch does not always quote the filename
This makes the diff --git header ambiguous again.
This PR finally adds handling for ambiguity in to parse patch
Fix#14812
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add in testing for no error
There is no way currently for CutDiffAroundLine in this test to cause an
error however, it should still be tested.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Reduce calls to git cat-file -s
There are multiple places where there are repeated calls to git cat-file
-s due to the blobs not being created with their size.
Through judicious use of git ls-tree -l and slight adjustments to the
indexer code we can avoid a lot of these calls.
* simplify by always expecting the long format
* Also always set the sized field and tell the indexer the update is sized
* Use cat-file --batch in GetLanguageStats
This PR moves to using a single cat-file --batch in GetLanguageStats
significantly reducing the number of processes spawned during language stat
processing.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* placate lint
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update modules/git/repo_language_stats_nogogit.go
Co-authored-by: a1012112796 <1012112796@qq.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: a1012112796 <1012112796@qq.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Somehow the test for detecting if branches are equal broke
this PR restores this functionality.
Fix#14502
Signed-off-by: Andrew Thornton <art27@cantab.net>
GetCommit() assumes that all 40 char strings are SHA1s. This leads to an
error if you try to do a PR on a branch which is 40 characters long.
This PR attempts the SHA first - and if it fails will switch to using rev-parse.
Fix#14470
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR improves HasPreviousCommit to prevent the automatic and recursive loading
of previous commits using git merge-base --is-ancestor and git rev-list
Fix#13684
Signed-off-by: Andrew Thornton <art27@cantab.net>
Memcached TTL cannot be > 30 days and if it is attempted the TTL is interpreted as
a unix timestamp.
This PR ensures that the TTL is switched to a unix timestamp in those cases.
Fix#14571
Signed-off-by: Andrew Thornton <art27@cantab.net>
Breaking the pipe is a valid way of killing a piped command and any error from
a broken cat-file batch command should be passed back up to the writer any way
therefore specifically logging it is unnecessary.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add pager to the branches page
* override pageSize if bigger than max
* Make branches commit range configurable
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: silverwind <me@silverwind.io>
* remove github.com/unknwon/com from models
* dont use "com.ToStr()"
* replace "com.ToStr" with "fmt.Sprint" where its easy to do
* more refactor
* fix test
* just "proxy" Copy func for now
* as per @lunny
* Fix git.parseTagData()
close#14092
* Add Test
* add message to test
* limit readers
* git tag -m trims and terminates with a newline
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
There is a slight bug in the commit_reader introduced in #13673 whereby
commit messages which have a final unterminated line miss their final line.
This PR fixes this.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move last commit cache back into modules/git
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Remove go-git from the interface for last commit cache
Signed-off-by: Andrew Thornton <art27@cantab.net>
* move cacheref to last_commit_cache
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Remove go-git from routers/private/hook
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move FindLFSFiles to pipeline
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Make no-go-git variants
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Submodule RefID
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix issue with GetCommitsInfo
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix GetLastCommitForPaths
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Improve efficiency
Signed-off-by: Andrew Thornton <art27@cantab.net>
* More efficiency
Signed-off-by: Andrew Thornton <art27@cantab.net>
* even faster
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Reduce duplication
* As per @lunny
Signed-off-by: Andrew Thornton <art27@cantab.net>
* attempt to fix drone
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix test-tags
Signed-off-by: Andrew Thornton <art27@cantab.net>
* default to use no-go-git variants and add gogit build tag
Signed-off-by: Andrew Thornton <art27@cantab.net>
* placate lint
Signed-off-by: Andrew Thornton <art27@cantab.net>
* as per @6543
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
After cancelling the context we still need to wait for the
command to finish otherwise zombie processes may occur
Fix#13987
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Gusted
Lunny Xiao
