Skip to content

Commit db611f4

Browse files
maobaolongmaobaolong
authored
[#436][FOLLOWUP] docs: Add documents for localStorageManagerClass and MultiPartLocalStorageManager (#2266)
### What changes were proposed in this pull request? Add config localStorageManagerClass description to introduce the default value and MultiPartLocalStorageManager. ### Why are the changes needed? Add missing docs. Fix: #436 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? No need.
1 parent ac89c19 commit db611f4

File tree

1 file changed

+8
-7
lines changed

1 file changed

+8
-7
lines changed

‎docs/server_guide.md

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -124,13 +124,14 @@ This document will introduce how to deploy Uniffle shuffle servers.
124124
| rss.server.blockIdManagerClass | org.apache.uniffle.server.block.DefaultShuffleBlockIdManager | The block id manager class. It is used to manage block id. |
125125

126126
### Advanced Configurations
127-
| Property Name | Default | Description |
128-
|--------------------------------------------------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
129-
| rss.server.storageMediaProvider.from.env.key | - | Sometimes, the local storage type/media info is provided by external system. RSS would read the env key defined by this configuration and get info about the storage media of its basePaths |
130-
| rss.server.decommission.check.interval | 60000 | The interval(ms) to check if all applications have finish when server is decommissioning |
131-
| rss.server.decommission.shutdown | true | Whether shutdown the server after server is decommissioned |
132-
| rss.server.health.checker.script.path | - | The health script path for `HealthScriptChecker`. To enable `HealthScriptChecker`, need to set `rss.server.health.checker.class.names` and set `rss.server.health.check.enable` to true. |
133-
| rss.server.health.checker.script.execute.timeout | 5000 | Timeout for `HealthScriptChecker` execute health script.(ms) |
127+
| Property Name | Default | Description |
128+
|--------------------------------------------------|-------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
129+
| rss.server.storageMediaProvider.from.env.key | - | Sometimes, the local storage type/media info is provided by external system. RSS would read the env key defined by this configuration and get info about the storage media of its basePaths |
130+
| rss.server.decommission.check.interval | 60000 | The interval(ms) to check if all applications have finish when server is decommissioning |
131+
| rss.server.decommission.shutdown | true | Whether shutdown the server after server is decommissioned |
132+
| rss.server.health.checker.script.path | - | The health script path for `HealthScriptChecker`. To enable `HealthScriptChecker`, need to set `rss.server.health.checker.class.names` and set `rss.server.health.check.enable` to true. |
133+
| rss.server.health.checker.script.execute.timeout | 5000 | Timeout for `HealthScriptChecker` execute health script.(ms) |
134+
| rss.server.localStorageManagerClass | org.apache.uniffle.server.storage.LocalStorageManager | The class for implementing local storage management can be set to `org.apache.uniffle.server.storage.MultiPartLocalStorageManager`. This class enables the storage of each partition across multiple disks in multiple parts. Please note that this configuration requires coordination with the client. Therefore, ensure that the client version is equal to or newer than the this version before upgrading the server. |
134135

135136
### Huge Partition Optimization
136137
A huge partition is a common problem for Spark/MR and so on, caused by data skew. And it can cause the shuffle server to become unstable. To solve this, we introduce some mechanisms to limit the writing of huge partitions to avoid affecting regular partitions, and introduce a hard limit config to reject extremely huge partition, more details can be found in [ISSUE-378](https://github.com/apache/incubator-uniffle/issues/378). The basic rules for limiting large partitions are memory usage limits and flushing individual buffers directly to persistent storage.

0 commit comments

Comments
 (0)