6f9c278559
# ⚠️ Breaking Many deprecated queue config options are removed (actually, they should have been removed in 1.18/1.19). If you see the fatal message when starting Gitea: "Please update your app.ini to remove deprecated config options", please follow the error messages to remove these options from your app.ini. Example: ``` 2023/05/06 19:39:22 [E] Removed queue option: `[indexer].ISSUE_INDEXER_QUEUE_TYPE`. Use new options in `[queue.issue_indexer]` 2023/05/06 19:39:22 [E] Removed queue option: `[indexer].UPDATE_BUFFER_LEN`. Use new options in `[queue.issue_indexer]` 2023/05/06 19:39:22 [F] Please update your app.ini to remove deprecated config options ``` Many options in `[queue]` are are dropped, including: `WRAP_IF_NECESSARY`, `MAX_ATTEMPTS`, `TIMEOUT`, `WORKERS`, `BLOCK_TIMEOUT`, `BOOST_TIMEOUT`, `BOOST_WORKERS`, they can be removed from app.ini. # The problem The old queue package has some legacy problems: * complexity: I doubt few people could tell how it works. * maintainability: Too many channels and mutex/cond are mixed together, too many different structs/interfaces depends each other. * stability: due to the complexity & maintainability, sometimes there are strange bugs and difficult to debug, and some code doesn't have test (indeed some code is difficult to test because a lot of things are mixed together). * general applicability: although it is called "queue", its behavior is not a well-known queue. * scalability: it doesn't seem easy to make it work with a cluster without breaking its behaviors. It came from some very old code to "avoid breaking", however, its technical debt is too heavy now. It's a good time to introduce a better "queue" package. # The new queue package It keeps using old config and concept as much as possible. * It only contains two major kinds of concepts: * The "base queue": channel, levelqueue, redis * They have the same abstraction, the same interface, and they are tested by the same testing code. * The "WokerPoolQueue", it uses the "base queue" to provide "worker pool" function, calls the "handler" to process the data in the base queue. * The new code doesn't do "PushBack" * Think about a queue with many workers, the "PushBack" can't guarantee the order for re-queued unhandled items, so in new code it just does "normal push" * The new code doesn't do "pause/resume" * The "pause/resume" was designed to handle some handler's failure: eg: document indexer (elasticsearch) is down * If a queue is paused for long time, either the producers blocks or the new items are dropped. * The new code doesn't do such "pause/resume" trick, it's not a common queue's behavior and it doesn't help much. * If there are unhandled items, the "push" function just blocks for a few seconds and then re-queue them and retry. * The new code doesn't do "worker booster" * Gitea's queue's handlers are light functions, the cost is only the go-routine, so it doesn't make sense to "boost" them. * The new code only use "max worker number" to limit the concurrent workers. * The new "Push" never blocks forever * Instead of creating more and more blocking goroutines, return an error is more friendly to the server and to the end user. There are more details in code comments: eg: the "Flush" problem, the strange "code.index" hanging problem, the "immediate" queue problem. Almost ready for review. TODO: * [x] add some necessary comments during review * [x] add some more tests if necessary * [x] update documents and config options * [x] test max worker / active worker * [x] re-run the CI tasks to see whether any test is flaky * [x] improve the `handleOldLengthConfiguration` to provide more friendly messages * [x] fine tune default config values (eg: length?) ## Code coverage: 
64 lines
2.6 KiB
Markdown
64 lines
2.6 KiB
Markdown
---
|
|
date: "2019-09-06T01:35:00-03:00"
|
|
title: "Repository indexer"
|
|
slug: "repo-indexer"
|
|
weight: 45
|
|
toc: false
|
|
draft: false
|
|
aliases:
|
|
- /en-us/repo-indexer
|
|
menu:
|
|
sidebar:
|
|
parent: "administration"
|
|
name: "Repository indexer"
|
|
weight: 45
|
|
identifier: "repo-indexer"
|
|
---
|
|
|
|
# Repository indexer
|
|
|
|
**Table of Contents**
|
|
|
|
{{< toc >}}
|
|
|
|
## Setting up the repository indexer
|
|
|
|
Gitea can search through the files of the repositories by enabling this function in your [`app.ini`](https://docs.gitea.io/en-us/config-cheat-sheet/):
|
|
|
|
```ini
|
|
[indexer]
|
|
; ...
|
|
REPO_INDEXER_ENABLED = true
|
|
REPO_INDEXER_PATH = indexers/repos.bleve
|
|
MAX_FILE_SIZE = 1048576
|
|
REPO_INDEXER_INCLUDE =
|
|
REPO_INDEXER_EXCLUDE = resources/bin/**
|
|
```
|
|
|
|
Please bear in mind that indexing the contents can consume a lot of system resources, especially when the index is created for the first time or globally updated (e.g. after upgrading Gitea).
|
|
|
|
### Choosing the files for indexing by size
|
|
|
|
The `MAX_FILE_SIZE` option will make the indexer skip all files larger than the specified value.
|
|
|
|
### Choosing the files for indexing by path
|
|
|
|
Gitea applies glob pattern matching from the [`gobwas/glob` library](https://github.com/gobwas/glob) to choose which files will be included in the index.
|
|
|
|
Limiting the list of files prevents the indexes from becoming polluted with derived or irrelevant files (e.g. lss, sym, map, etc.), so the search results are more relevant. It can also help reduce the index size.
|
|
|
|
`REPO_INDEXER_EXCLUDE_VENDORED` (default: true) excludes vendored files from index.
|
|
|
|
`REPO_INDEXER_INCLUDE` (default: empty) is a comma separated list of glob patterns to **include** in the index. An empty list means "_include all files_".
|
|
`REPO_INDEXER_EXCLUDE` (default: empty) is a comma separated list of glob patterns to **exclude** from the index. Files that match this list will not be indexed. `REPO_INDEXER_EXCLUDE` takes precedence over `REPO_INDEXER_INCLUDE`.
|
|
|
|
Pattern matching works as follows:
|
|
|
|
- To match all files with a `.txt` extension no matter what directory, use `**.txt`.
|
|
- To match all files with a `.txt` extension _only at the root level of the repository_, use `*.txt`.
|
|
- To match all files inside `resources/bin` and below, use `resources/bin/**`.
|
|
- To match all files _immediately inside_ `resources/bin`, use `resources/bin/*`.
|
|
- To match all files named `Makefile`, use `**Makefile`.
|
|
- Matching a directory has no effect; the pattern `resources/bin` will not include/exclude files inside that directory; `resources/bin/**` will.
|
|
- All files and patterns are normalized to lower case, so `**Makefile`, `**makefile` and `**MAKEFILE` are equivalent.
|