This PR contains multiple fixes. The most important of which is:
* Prevent hang in git cat-file if the repository is not a valid repository
Unfortunately it appears that if git cat-file is run in an invalid
repository it will hang until stdin is closed. This will result in
deadlocked /pulls pages and dangling git cat-file calls if a broken
repository is tried to be reviewed or pulls exists for a broken
repository.
Fix#14734Fix#9271Fix#16113
Otherwise there are a few small other fixes included which this PR was initially intending to fix:
* Fix panic on partial compares due to missing PullRequestWorkInProgressPrefixes
* Fix links on pulls pages due to regression from #17551 - by making most /issues routes match /pulls too - Fix#17983
* Fix links on feeds pages due to another regression from #17551 but also fix issue with syncing tags - Fix#17943
* Add missing locale entries for oauth group claims
* Prevent NPEs if ColorFormat is called on nil users, repos or teams.
* Add missing `X-Total-Count` and fix some related bugs
Adds `X-Total-Count` header to APIs that return a list but doesn't have it yet.
Fixed bugs:
* not returned after reporting error (39eb82446c/routers/api/v1/user/star.go (L70))
* crash with index out of bounds, API issue/issueSubscriptions
I also found various endpoints that return lists but do not apply/support pagination yet:
```
/repos/{owner}/{repo}/issues/{index}/labels
/repos/{owner}/{repo}/issues/comments/{id}/reactions
/repos/{owner}/{repo}/branch_protections
/repos/{owner}/{repo}/contents
/repos/{owner}/{repo}/hooks/git
/repos/{owner}/{repo}/issue_templates
/repos/{owner}/{repo}/releases/{id}/assets
/repos/{owner}/{repo}/reviewers
/repos/{owner}/{repo}/teams
/user/emails
/users/{username}/heatmap
```
If this is not expected, an new issue should be opened.
Closes#13043
* fmt
* Update routers/api/v1/repo/issue_subscription.go
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
* Use FindAndCount
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Co-authored-by: 6543 <6543@obermui.de>
* Add setting to OAuth handlers to override local 2FA settings
This PR adds a setting to OAuth and OpenID login sources to allow the source to
override local 2FA requirements.
Fix#13939
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix regression from #16544
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add scopes settings
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix trace logging in auth_openid
Signed-off-by: Andrew Thornton <art27@cantab.net>
* add required claim options
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move UpdateExternalUser to externalaccount
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Allow OAuth2/OIDC to set Admin/Restricted status
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Allow use of the same group claim name for the prohibit login value
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fixup! Move UpdateExternalUser to externalaccount
* as per wxiaoguang
Signed-off-by: Andrew Thornton <art27@cantab.net>
* add label back in
Signed-off-by: Andrew Thornton <art27@cantab.net>
* adjust localisation
Signed-off-by: Andrew Thornton <art27@cantab.net>
* placate lint
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Running `make test-backend` will delete `data/` due to reloading the configuration and resetting the appdatapath.
This PR removes this unnecessary config reload but also adds extra code in to the unittest main to prevent its cleanup from deleting the wrong directory.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Move keys to models/keys
* Rename models/keys -> models/asymkey
* change the missed package name
* Fix package alias
* Fix test
* Fix docs
* Fix test
* Fix test
* merge
* Some refactors related repository model
* Move more methods out of repository
* Move repository into models/repo
* Fix test
* Fix test
* some improvements
* Remove unnecessary function
The current implementation of checkBranchName is highly inefficient
involving opening the repository, the listing all of the branch names
checking them individually before then using using opened repo to get
the tags.
This PR avoids this by simply walking the references from show-ref
instead of opening the repository (in the nogogit case).
Signed-off-by: Andrew Thornton <art27@cantab.net>
* allways set a message-id on mails
* Add unit tests for mailer & Message-ID
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Improve install code to avoid low-level mistakes.
If a user tries to do a re-install in a Gitea database, they gets a warning and double check.
When Gitea runs, it never create empty app.ini automatically.
Also some small (related) refactoring:
* Refactor db.InitEngine related logic make it more clean (especially for the install code)
* Move some i18n strings out from setting.go to make the setting.go can be easily maintained.
* Show errors in CLI code if an incorrect app.ini is used.
* APP_DATA_PATH is created when installing, and checked when starting (no empty directory is created any more).
This PR registers requests with the process manager and manages hierarchy within the processes.
Git repos are then associated with a context, (usually the request's context) - with sub commands using this context as their base context.
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR adds another option to app.ini make co-committed-by and co-authored-by trailers
optional on a per server basis.
Fix#17194
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Remove unnecessary functions of User struct
* Move more database methods out of user struct
* Move more database methods out of user struct
* Fix template failure
* Fix bug
* Remove finished FIXME
* remove unnecessary code
* Improvements to content history
* initialize content history when making an edit to an old item created before the introduction of content history
* show edit history for code comments on pull request files tab
* Fix a flaw in keepLimitedContentHistory
Fix a flaw in keepLimitedContentHistory, the first and the last should never be deleted
* Remove obsolete eager initialization of content history
* Allow Loading of Diffs that are too large
This PR allows the loading of diffs that are suppressed because the file
is too large. It does not handle diffs of files which have lines which
are too long.
Fix#17738
Signed-off-by: Andrew Thornton <art27@cantab.net>
Unfortunately due to a misread on my behalf I missed that git diff only learned
--skip-to in version 2.31.0. Thus this functionality was not working on older versions
of git.
This PR adds a handler that simply allows for us to skip reading the diffs until
we find the correct file to skip to.
Fix#17731
Signed-off-by: Andrew Thornton <art27@cantab.net>
Use hostmacher to replace matchlist.
And we introduce a better DialContext to do a full host/IP check, otherwise the attackers can still bypass the allow/block list by a 302 redirection.
* Add settings to allow different SMTP envelope from address
Sometimes it may be advisable to hide or alias the from address on an SMTP mail
envelope. This PR adds two new options to the mailer to allow setting of an overriding
from address.
Fix#17477
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Use a standalone struct name for Organization
* recover unnecessary change
* make the code readable
* Fix template failure
* Fix template failure
* Move HasMemberWithUserID to org
* Fix test
* Remove unnecessary user type check
* Fix test
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Use check attribute code to check the assigned language of a file and send that in to
chroma as a hint for the language of the file.
Signed-off-by: Andrew Thornton <art27@cantab.net>
There are multiple places where Gitea does not properly escape URLs that it is building and there are multiple places where it builds urls when there is already a simpler function available to use this.
This is an extensive PR attempting to fix these issues.
1. The first commit in this PR looks through all href, src and links in the Gitea codebase and has attempted to catch all the places where there is potentially incomplete escaping.
2. Whilst doing this we will prefer to use functions that create URLs over recreating them by hand.
3. All uses of strings should be directly escaped - even if they are not currently expected to contain escaping characters. The main benefit to doing this will be that we can consider relaxing the constraints on user names and reponames in future.
4. The next commit looks at escaping in the wiki and re-considers the urls that are used there. Using the improved escaping here wiki files containing '/'. (This implementation will currently still place all of the wiki files the root directory of the repo but this would not be difficult to change.)
5. The title generation in feeds is now properly escaped.
6. EscapePound is no longer needed - urls should be PathEscaped / QueryEscaped as necessary but then re-escaped with Escape when creating html with locales Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
- Partialy resolvess #17596
- Resolves `badCall` errors from go-critic `badCall: suspicious Join on
1 argument`
- When only 1 argument is passed into `filepath.Join`, it won't do
anything special other than `filepath.Clean(...)` will be applied over
it.
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
* Simplify Gothic to use our session store instead of creating a different store
We have been using xormstore to provide a separate session store for our OAuth2 logins
however, this relies on using gorilla context and some doubling of our session storing.
We can however, simplify and simply use our own chi-based session store. Thus removing
a cookie and some of the weirdness with missing contexts.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* as per review
Signed-off-by: Andrew Thornton <art27@cantab.net>
* as per review
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Handle MaxTokenLength
Signed-off-by: Andrew Thornton <art27@cantab.net>
* oops
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
* Fixes#16558 CSV delimiter determiner
* Fixes#16558 - properly determine CSV delmiiter
* Moves quoteString to a new function
* Adds big test with lots of commas for tab delimited csv
* Adds comments
* Shortens the text of the test
* Removes single quotes from regexp as only double quotes need to be searched
* Fixes spelling
* Fixes check of length as it probalby will only be 1e4, not greater
* Makes sample size a const, properly removes truncated line
* Makes sample size a const, properly removes truncated line
* Fixes comment
* Fixes comment
* tests for FormatError() function
* Adds logic to find the limiter before or after a quoted value
* Simplifies regex
* Error tests
* Error tests
* Update modules/csv/csv.go
Co-authored-by: delvh <dev.lh@web.de>
* Update modules/csv/csv.go
Co-authored-by: delvh <dev.lh@web.de>
* Adds comments
* Update modules/csv/csv.go
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
* Add groups scope/claim to OICD/OAuth2
Add support for groups claim as part of the OIDC/OAuth2 flow.
Groups is a list of "org" and "org:team" strings to allow clients to
authorize based on the groups a user is part of.
Signed-off-by: Nico Schieder <code@nico-schieder.de>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Handle broken references in mirror sync
If there are broken references during a mirror attempt to fix using `git remote prune`.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Fixes#16837 if a column is deleted.
We were clobbering the columns that were added by looping through the aline (base) and then when bline (head) was looped through, it clobbered what was in the "cells" array that is show in the diff, and then left a nil cell because nothing was shifted.
This fix properly shifts the cells, and properly puts the b cell either at its location or after, according to what the aline placed in the cells.
This includes test, adding a new test function since adding/removing cells works best with three columns, not two, which results in 4 columns of the resulting cells because it has a deleted column and an added column. If you try this locally, you can try those cases and others, such as adding a column.
There was no need to do anything special for the rows when `aline == 0 || bline == 0` so that was removed. This allows the same code to be used for removed or added lines, with the bcell text always being the RightCell, acell text being the LeftCell.
I still added the patch zeripath gave at https://github.com/go-gitea/gitea/issues/16837#issuecomment-913007382 so that just in case for some reason a cell is nil (which shouldn't happen now) it doesn't throw a 500 error, so the user can at least view the raw diff.
Also fixes in the [view.go](https://github.com/go-gitea/gitea/pull/17018/files#diff-43a7f4747c7ba8bff888c9be11affaafd595fd55d27f3333840eb19df9fad393L521) file how if a CSV file is empty (either created empty or if you edit it and remove all contents) it throws a huge 500 error when you then save it (when you view the file). Since we allow creating, saving and pushing empty files, we shouldn't throw an error on an empty CSV file, but just show its empty contents. This doesn't happen if it is a Markdown file or other type of file that is empty.
EDIT: Now handled in the markup/csv renderer code
Convert the old mirror syncing queue to the more modern queue format.
Fix a bug in the from the repo-archive queue PR - the assumption was made that uniqueness could be enforced with by checking equality in a map in channel unique queues - however this only works for primitive types - which was the initial intention but is an imperfect. This is fixed by marshalling the data and placing the martialled data in the unique map instead.
The documentation is also updated to add information about the deprecated configuration values.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* issue content history
* Use timeutil.TimeStampNow() for content history time instead of issue/comment.UpdatedUnix (which are not updated in time)
* i18n for frontend
* refactor
* clean up
* fix refactor
* re-format
* temp refactor
* follow db refactor
* rename IssueContentHistory to ContentHistory, remove empty model tags
* fix html
* use avatar refactor to generate avatar url
* add unit test, keep at most 20 history revisions.
* re-format
* syntax nit
* Add issue content history table
* Update models/migrations/v197.go
Co-authored-by: 6543 <6543@obermui.de>
* fix merge
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
- Update default branch if needed
- Update protected branch if needed
- Update all not merged pull request base branch name
- Rename git branch
- Record this rename work and auto redirect for old branch on ui
Signed-off-by: a1012112796 <1012112796@qq.com>
Co-authored-by: delvh <dev.lh@web.de>
Why this refactor
The goal is to move most files from `models` package to `models.xxx` package. Many models depend on avatar model, so just move this first.
And the existing logic is not clear, there are too many function like `AvatarLink`, `RelAvatarLink`, `SizedRelAvatarLink`, `SizedAvatarLink`, `MakeFinalAvatarURL`, `HashedAvatarLink`, etc. This refactor make everything clear:
* user.AvatarLink()
* user.AvatarLinkWithSize(size)
* avatars.GenerateEmailAvatarFastLink(email, size)
* avatars.GenerateEmailAvatarFinalLink(email, size)
And many duplicated code are deleted in route handler, the handler and the model share the same avatar logic now.
There was a mistake in the template file: `templates/mail/issue/assigned.tmpl`
where the repourl was generated from a non-existent release instead of the issue.
This PR changes this to use the issue but also ensure that the issue repo is loaded.
It also slightly improves the English locale string.
Fix#17160
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
This PR changes the compare page to make the "..." in the between branches a clickable
link. This changes the comparison type from "..." to "..". Similarly it makes the
initial compare icon clickable to switch the head and base branches.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Allow LDAP Sources to provide Avatars
Add setting to LDAP source to allow it to provide an Avatar.
Currently this is required to point to the image bytes.
Fix#4144
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Rename as Avatar Attribute (drop JPEG)
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Always synchronize avatar if there is change
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Actually get the avatar from the ldap
Signed-off-by: Andrew Thornton <art27@cantab.net>
* clean-up
Signed-off-by: Andrew Thornton <art27@cantab.net>
* use len()>0 rather than != ""
Signed-off-by: Andrew Thornton <art27@cantab.net>
* slight shortcut in IsUploadAvatarChanged
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Add SkipLocal2FA option to other pam and smtp sources
Extend #16954 to allow setting skip local 2fa on pam and SMTP authentication sources
Signed-off-by: Andrew Thornton <art27@cantab.net>
* make SkipLocal2FA omitempty
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
* DBContext is just a Context
This PR removes some of the specialness from the DBContext and makes it context
This allows us to simplify the GetEngine code to wrap around any context in future
and means that we can change our loadRepo(e Engine) functions to simply take contexts.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* fix unit tests
Signed-off-by: Andrew Thornton <art27@cantab.net>
* another place that needs to set the initial context
Signed-off-by: Andrew Thornton <art27@cantab.net>
* avoid race
Signed-off-by: Andrew Thornton <art27@cantab.net>
* change attachment error
Signed-off-by: Andrew Thornton <art27@cantab.net>
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
https://developers.dingtalk.com/document/app/message-link-description
To open the link in browser, we should use this URL: `"dingtalk://dingtalkclient/page/link?pc_slide=false&url=" + url.QueryEscape(singleURL)`
Otherwise the page is displayed inside DingTalk client, it makes users very difficult to visit non-public URLs in DingTalk webhook messages.
This PR extends #16594 to allow LDAP to be able to be set to skip local 2FA too. The technique used here would be extensible to PAM and SMTP sources.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Fixes#16381
Note that changes to unprotected files via the web editor still cannot be pushed directly to the protected branch. I could easily add such support for edits and deletes if needed. But for adding, uploading or renaming unprotected files, it is not trivial.
* Extract & Move GetAffectedFiles to modules/git
If there is an error when performing the admin or restricted filter search
ensure that the filter and the userDN are reported.
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR adds a setting to OAuth and OpenID login sources to allow the source to
skip local 2FA authentication.
Fix#13939
Signed-off-by: Andrew Thornton <art27@cantab.net>
Replaces #16262
Replaces #16250
Replaces #14833
This PR first implements a `git check-attr` pipe reader - using `git check-attr --stdin -z --cached` - taking account of the change in the output format in git 1.8.5 and creates a helper function to read a tree into a temporary index file for that pipe reader.
It then wires this in to the language stats helper and into the git diff generation.
Files which are marked generated will be folded by default.
Fixes#14786Fixes#12653
When create a new issue or comment and paste/upload an attachment/image, it will not assign an issue id before submit. So if user give up the creating, the attachments will lost key feature and become dirty content. We don't know if we need to delete the attachment even if the repository deleted.
This PR add a repo_id in attachment table so that even if a new upload attachment with no issue_id or release_id but should have repo_id. When deleting a repository, they could also be deleted.
Co-authored-by: 6543 <6543@obermui.de>
Rename`[repository]` `DISABLE_MIRRORS` to `[mirror]` `DISABLE_NEW_PULL` and add `ENABLED` and `DISABLE_NEW_PUSH` with the below meanings:
- `ENABLED`: **true**: Enables the mirror functionality. Set to **false** to disable all mirrors.
- `DISABLE_NEW_PULL`: **false**: Disable the creation of **new** mirrors. Pre-existing mirrors remain valid.
- `DISABLE_NEW_PUSH`: **false**: Disable the creation of **new** push mirrors. Pre-existing mirrors remain valid.
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
The lastLeftIdx should be reset at the same time as creating a new section otherwise
it is possible for a second addition to end up attempting to read a nil entry.
Fix#16943
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix dump and restore
* return different error message for get commit
* Fix missing delete release attachment when deleting repository
* Fix ci and add some comments
Co-authored-by: zeripath <art27@cantab.net>
Lunny Xiao
zeripath
