Manage Configuration#
Note
这个项目的配置管理功能由 config_patterns Python 库提供. 建议你先阅读该项目的文档, 有一个大该的了解既可.
Declare Configuration Schema#
每个 Environment 下会有多个 Server (environment 和 server 的概念请参考 这篇文档). 一个 Server 的详细配置是在 acore_server_config/config/define/server.py 模块中被定义的. Server
类的实例代表了一个 Server 的配置数据. 下面是该模块的源码:
acore_server_config/config/define/server.py
1# -*- coding: utf-8 -*-
2
3"""
4todo: doc string
5"""
6
7import typing as T
8import dataclasses
9
10from acore_constants.api import ServerLifeCycle
11
12
13@dataclasses.dataclass
14class Server:
15 """
16 Per Game Server configuration.
17
18 :param id: Server id, the naming convention is ``${env_name}-${server_name}``.
19 :param ec2_ami_id: the AMI id for the game server.
20 :param ec2_instance_type: the EC2 instance type for the game server.
21 :param ec2_subnet_id: the EC2 subnet id for the game server.
22 :param ec2_key_name: the EC2 ssh key name for the game server.
23 :param ec2_eip_allocation_id: if you need a static IP, then create
24 an Elastic IP address and put the allocation id here. otherwise,
25 use the automatic public IP address.
26 :param acore_soap_app_version: the acore_soap_app-project git tag for bootstrap.
27 :param acore_server_bootstrap_version: the acore_server_bootstrap-project
28 git tag for bootstrap.
29 :param db_snapshot_id: the snapshot id to create the RDS DB instance.
30 :param db_instance_class: the RDS instance class for the game database.
31 :param db_engine_version: the RDS engine version (all the way to minor).
32 :param db_admin_username: the RDS admin username, usually this is admin.
33 :param db_admin_password: the RDS admin password, we need this password.
34 to create the database user for game server.
35 :param db_username: the database user for game server.
36 :param db_password: the database password for game server.
37 :param lifecycle: the logic "game server (both EC2 and RDS)" lifecycle definition.
38 :param authserver_conf: custom config for authserver.conf.
39 :param worldserver_conf: custom config for worldserver.conf.
40 :param mod_lua_engine_conf: custom config for mod_LuaEngine.conf.
41 """
42
43 id: T.Optional[str] = dataclasses.field(default=None)
44 # EC2 related
45 ec2_ami_id: T.Optional[str] = dataclasses.field(default=None)
46 ec2_instance_type: T.Optional[str] = dataclasses.field(default=None)
47 ec2_subnet_id: T.Optional[str] = dataclasses.field(default=None)
48 ec2_key_name: T.Optional[str] = dataclasses.field(default=None)
49 ec2_eip_allocation_id: T.Optional[str] = dataclasses.field(default=None)
50 acore_soap_app_version: T.Optional[str] = dataclasses.field(default=None)
51 acore_db_app_version: T.Optional[str] = dataclasses.field(default=None)
52 acore_server_bootstrap_version: T.Optional[str] = dataclasses.field(default=None)
53 # RDS related
54 db_snapshot_id: T.Optional[str] = dataclasses.field(default=None)
55 db_instance_class: T.Optional[str] = dataclasses.field(default=None)
56 db_engine_version: T.Optional[str] = dataclasses.field(default=None)
57 db_admin_username: T.Optional[str] = dataclasses.field(default=None)
58 db_admin_password: T.Optional[str] = dataclasses.field(default=None)
59 db_username: T.Optional[str] = dataclasses.field(default=None)
60 db_password: T.Optional[str] = dataclasses.field(default=None)
61 # EC2 and RDS related
62 lifecycle: T.Optional[str] = dataclasses.field(default=None)
63 # authserver.conf, worldserver.conf, ...
64 authserver_conf: T.Dict[str, str] = dataclasses.field(default_factory=dict)
65 worldserver_conf: T.Dict[str, str] = dataclasses.field(default_factory=dict)
66 mod_lua_engine_conf: T.Dict[str, str] = dataclasses.field(default_factory=dict)
67
68 def __post_init__(self):
69 if self.lifecycle not in [
70 ServerLifeCycle.running,
71 ServerLifeCycle.smart_running,
72 ServerLifeCycle.stopped,
73 ServerLifeCycle.deleted,
74 ]: # pragma: no cover
75 raise ValueError(f"{self.lifecycle!r} is not a valid lifecycle definition!")
76
77 def is_ready_for_create_new_server(self) -> bool:
78 """
79 Check if the configuration is sufficient for creating new server.
80
81 See: https://acore-server.readthedocs.io/en/latest/search.html?q=Operation+and+Workflow&check_keywords=yes&area=default#
82 """
83 not_none_fields = [
84 "id",
85 "ec2_ami_id",
86 "ec2_instance_type",
87 "ec2_subnet_id",
88 "ec2_key_name",
89 "acore_soap_app_version",
90 "acore_db_app_version",
91 "acore_server_bootstrap_version",
92 "db_instance_class",
93 "db_engine_version",
94 "db_admin_username",
95 "db_admin_password",
96 "db_username",
97 "db_password",
98 ]
99 for field in not_none_fields:
100 if getattr(self, field) is None:
101 return False
102 return True
103
104 def is_ready_for_create_cloned_server(self) -> bool:
105 """
106 Check if the configuration is sufficient for creating cloned server.
107
108 See: https://acore-server.readthedocs.io/en/latest/search.html?q=Operation+and+Workflow&check_keywords=yes&area=default#
109 """
110 not_none_fields = [
111 "id",
112 "ec2_instance_type",
113 "ec2_subnet_id",
114 "ec2_key_name",
115 "acore_soap_app_version",
116 "acore_db_app_version",
117 "acore_server_bootstrap_version",
118 "db_instance_class",
119 # "db_engine_version", # clone 的时候不需要指定 engine version, 会自动继承
120 # "db_admin_username", # clone 的时候不需要指定 admin username, 会自动继承
121 "db_admin_password",
122 "db_username",
123 "db_password",
124 ]
125 for field in not_none_fields:
126 if getattr(self, field) is None:
127 return False
128 return True
129
130 def is_ready_for_create_updated_server(self) -> bool:
131 """
132 Check if the configuration is sufficient for creating updated server.
133
134 See: https://acore-server.readthedocs.io/en/latest/search.html?q=Operation+and+Workflow&check_keywords=yes&area=default#
135 """
136 not_none_fields = [
137 "id",
138 "ec2_ami_id",
139 "ec2_instance_type",
140 "ec2_subnet_id",
141 "ec2_key_name",
142 "acore_soap_app_version",
143 "acore_db_app_version",
144 "acore_server_bootstrap_version",
145 "db_instance_class",
146 # "db_engine_version", # update server 的时候不需要指定 engine version, 因为我们会用已经存在的数据库
147 # "db_admin_username", # update server 的时候不需要指定 admin username, 因为我们会用已经存在的数据库
148 "db_admin_password",
149 "db_username",
150 "db_password",
151 ]
152 for field in not_none_fields:
153 if getattr(self, field) is None:
154 return False
155 return True
156
157 def is_ready_for_stop_server(self) -> bool:
158 """
159 Check if the configuration is sufficient for stopping server.
160
161 See: https://acore-server.readthedocs.io/en/latest/search.html?q=Operation+and+Workflow&check_keywords=yes&area=default#
162 """
163 not_none_fields = [
164 "id",
165 ]
166 for field in not_none_fields:
167 if getattr(self, field) is None:
168 return False
169 return True
170
171 def is_ready_for_start_server(self) -> bool:
172 """
173 Check if the configuration is sufficient for starting server.
174
175 See: https://acore-server.readthedocs.io/en/latest/search.html?q=Operation+and+Workflow&check_keywords=yes&area=default#
176 """
177 not_none_fields = [
178 "id",
179 ]
180 for field in not_none_fields:
181 if getattr(self, field) is None:
182 return False
183 return True
184
185 def is_ready_for_delete_server(self) -> bool:
186 """
187 Check if the configuration is sufficient for deleting server.
188
189 See: https://acore-server.readthedocs.io/en/latest/search.html?q=Operation+and+Workflow&check_keywords=yes&area=default#
190 """
191 not_none_fields = [
192 "id",
193 ]
194 for field in not_none_fields:
195 if getattr(self, field) is None:
196 return False
197 return True
198
199
200@dataclasses.dataclass
201class ServerMixin:
202 servers: T.Dict[str, Server] = dataclasses.field(default_factory=dict)
203
204 @property
205 def server_blue(self) -> Server:
206 return self.servers["blue"]
207
208 @property
209 def server_green(self) -> Server:
210 return self.servers["green"]
211
212 @property
213 def server_black(self) -> Server:
214 return self.servers["black"]
215
216 @property
217 def server_white(self) -> Server:
218 return self.servers["white"]
219
220 @property
221 def server_yellow(self) -> Server:
222 return self.servers["yello"]
223
224 @property
225 def server_orange(self) -> Server:
226 return self.servers["orange"]
Local Configuration Data#
配置数据的源头是由管理员在本地编写好数据文件然后部署到 AWS S3 中的. 本地的配置文件分两个, 普通配置数据的 config/config.json, 和敏感配置数据的 ${HOME}/.projects/acore_server_config/config-secret.json
文件 (这是一个本地路径). 普通配置数据文件会被 check in 到 Git 中, 而敏感配置数据不会.
而读取 Configuration Data 会有两种情况:
管理员在本地开发时从本地数据文件中读取数据.
游戏服务器在 EC2 上运行程序并读取数据.
下面我们分情况来介绍.
1. Load Configuration Data From Local Files#
acore_server_config/config/init.py 模块实现了从本地文件中读取配置数据的功能. 你只要运行 from acore_server_config.config.init import config
既可获得一个 config 对象. 该模块的源码如下:
acore_server_config/config/init.py
1# -*- coding: utf-8 -*-
2
3"""
4注, 这个模块不属于公开 API 的一部分, 仅仅用于在 acore_server_config 项目本身内部用于部署
5配置数据. 如果你在其他项目中引用了这个项目, 请使用 acore_server_config.api 模块中的内容.
6"""
7
8import json
9
10from ..paths import path_config_json, path_config_secret_json
11from ..runtime import IS_LOCAL
12
13from .define import EnvEnum, Env, Config
14
15if IS_LOCAL:
16 # ensure that the config-secret.json file exists
17 # it should be at the ${HOME}/.projects/wserver_infra/config-secret.json
18 # this code block is only used to onboard first time user of this
19 # project template. Once you know about how to handle the config-secret.json file,
20 # you can delete this code block.
21 if not path_config_secret_json.exists(): # pragma: no cover
22 path_config_secret_json.parent.mkdir(parents=True, exist_ok=True)
23 path_config_secret_json.write_text(
24 json.dumps(
25 {
26 "_shared": {},
27 EnvEnum.sbx.value: {"password": f"{EnvEnum.sbx.value}.password"},
28 EnvEnum.tst.value: {"password": f"{EnvEnum.tst.value}.password"},
29 EnvEnum.prd.value: {"password": f"{EnvEnum.prd.value}.password"},
30 },
31 indent=4,
32 )
33 )
34
35 # read non-sensitive config and sensitive config from local file system
36 config = Config.read(
37 env_class=Env,
38 env_enum_class=EnvEnum,
39 path_config=path_config_json,
40 path_secret_config=path_config_secret_json,
41 )
42else:
43 raise NotImplementedError
2. Load Configuration Data From AWS S3 on EC2#
acore_server_config/config/loader.py 模块实现了从 AWS S3 中读取配置数据的功能. 它有两个关键的类:
ConfigLoader
: 让管理员从 AWS S3 上读取配置数据. 管理员在部署了配置数据后可以用这个类从 AWS S3 将其读回来. 该操作常用于 Debug.Ec2ConfigLoader
: 让运行在 EC2 上的脚本自己发现 (自省) 自己是哪个服务器, 然后到对应的 AWS S3 object 中读取属于自己的配置数据. 游戏服务器启动时的自动化脚本就会用到这个类.
下面是该模块的源码:
acore_server_config/config/loader.py
1# -*- coding: utf-8 -*-
2
3"""
4该模块为使用 ``acore_server_config`` 库的外部项目提供了一套能读取服务器 config 的接口.
5相比之下 ``acore_server_config.config.init`` 模块是为 ``acore_server_config`` 库
6内部用来从本地文件系统读取配置数据的, 外部项目不应该使用它.
7
8该模块有两个 Public API:
9
10- :class:`Ec2ConfigLoader`: 用于在 EC2 上运行脚本, 用 "自省" 的方式获得自己的配置数据.
11- :class:`ConfigLoader`: 用于在任意其他环境显式的加载配置数据.
12"""
13
14import dataclasses
15import typing as T
16
17from s3pathlib import S3Path
18from simple_aws_ec2.api import Ec2Instance
19from acore_constants.api import TagKey
20
21from ..boto_ses import bsm as default_bsm
22
23from .define import EnvEnum, Env, Config, Server
24
25
26if T.TYPE_CHECKING: # pragma: no cover
27 from boto_session_manager import BotoSesManager
28
29
30def _get_default_s3folder_config(bsm: "BotoSesManager") -> str:
31 """
32 获得默认的 S3 配置数据的根目录.
33
34 .. versionchanged:: 0.6.3
35
36 Change the default config bucket name from
37 ``{bsm.aws_account_id}-{bsm.aws_region}-artifacts``
38 to ``{bsm.aws_account_alias}-{bsm.aws_region}-artifacts``.
39 """
40 return (
41 S3Path(f"s3://{bsm.aws_account_alias}-{bsm.aws_region}-artifacts")
42 .joinpath(
43 "projects",
44 "acore_server_config",
45 "config",
46 )
47 .to_dir()
48 ).uri
49
50
51def _get_default_parameter_name_prefix() -> str:
52 """
53 获得默认的 AWS Parameter Store 的参数名前缀. 这取决于我们 deploy 的时候用的前缀是什么.
54 """
55 return "acore_server_config"
56
57
58def get_this_server_id(bsm: "BotoSesManager") -> str: # pragma: no cover
59 """
60 在 EC2 上通过 "自省", 获得这个服务器的 server_id. 它的 naming convention 是
61 ``${env_name}-${server_name}``.
62 """
63 ec2_inst = Ec2Instance.from_ec2_inside(bsm.ec2_client)
64 server_id = ec2_inst.tags[TagKey.SERVER_ID]
65 return server_id
66
67
68def parse_server_id(server_id: str) -> T.Tuple[str, str]:
69 """
70 解析 server_id, 返回 (env_name, server_name) 的 tuple.
71 """
72 env_name, server_name = server_id.split("-", 1)
73 return env_name, server_name
74
75
76def get_config(
77 bsm: "BotoSesManager" = default_bsm,
78 parameter_name_prefix: T.Optional[str] = None,
79 env_name: T.Optional[str] = None,
80 s3folder_config: T.Optional[str] = None,
81) -> Config:
82 """
83 获取这个 ``Config`` 对象的数据. 详细的数据结构请参考
84 :class:`acore_server_config.config.main.Config`.
85
86 :param bsm: BotoSesManager 实例.
87 :param parameter_name_prefix: the parameter name prefix, the full name will
88 be ${parameter_name_prefix}-${env_name}.
89 :param env_name: the environment name of the env specific config you want to
90 load from, stx, tst, prd, etc. If None, then load the master config.
91 :param s3folder_config: S3 配置数据的根目录, 默认为
92 s3://aws_account_id}-{aws_region}-artifacts/projects/acore_server_config/config/
93 """
94 if parameter_name_prefix is None:
95 parameter_name_prefix = _get_default_parameter_name_prefix()
96 if env_name is None: # pragma: no cover
97 parameter_name = parameter_name_prefix
98 else:
99 parameter_name = f"{parameter_name_prefix}-{env_name}"
100
101 if s3folder_config is None:
102 s3folder_config = _get_default_s3folder_config(bsm=bsm)
103
104 config = Config.read(
105 env_class=Env,
106 env_enum_class=EnvEnum,
107 bsm=bsm,
108 parameter_name=parameter_name,
109 s3folder_config=s3folder_config,
110 )
111
112 return config
113
114
115# [ec2configloader-start]
116@dataclasses.dataclass
117class Ec2ConfigLoader:
118 """
119 用于在 EC2 上运行脚本, 用 "自省" 的方式获得自己的配置数据. 开始时请使用
120 :meth:`Ec2ConfigLoader.load` 方法获得当前 EC2 的配置数据.
121
122 用法:
123
124 .. code-block:: python
125
126 >>> server = Ec2ConfigLoader.load(...)
127 >>> server
128 Server(id='sbx-blue', db_admin_password='sbx*dummy4test', db_username='myuser', db_password='sbx*dummy4test')
129 """
130
131 @classmethod
132 def load(
133 cls,
134 parameter_name_prefix: T.Optional[str] = None,
135 s3folder_config: T.Optional[str] = None,
136 server_id: T.Optional[str] = None,
137 bsm: "BotoSesManager" = default_bsm,
138 ) -> Server:
139 """
140 获得当前 EC2 的配置数据, 返回一个
141 :class:`~acore_server_config.config.define.server.Server` 对象.
142
143 :param parameter_name_prefix: the parameter name prefix, the full name will
144 be ${parameter_name_prefix}-${env_name}.
145 :param s3folder_config: S3 配置数据的根目录, 默认为
146 s3://aws_account_id}-{aws_region}-artifacts/projects/acore_server_config/config/
147 :param server_id: 强制指定 server_id, 跳过 "自省" 阶段. 常用于测试. 这个 server_id
148 的格式为: ${env_name}-${server_name}, 例如: sbx-blue
149 :param bsm: ``boto_session_manager.BotoSesManager`` object, if not provided,
150 then use the current runtime default AWS CLI profile.
151 """
152 if server_id is None: # pragma: no cover
153 server_id = get_this_server_id(bsm=bsm)
154 env_name, server_name = parse_server_id(server_id=server_id)
155 config = get_config(
156 bsm=bsm,
157 parameter_name_prefix=parameter_name_prefix,
158 env_name=env_name,
159 s3folder_config=s3folder_config,
160 )
161 env = config.get_env(env_name)
162 return env.servers[server_name]
163# [ec2configloader-end]
164
165
166# [configloader-start]
167@dataclasses.dataclass
168class ConfigLoader:
169 """
170 用于在任意其他环境显式的加载配置数据. 开始时请使用 :meth:`ConfigLoader.new` 方法创建
171 一个新的 Loader 对象, 将配置数据加载到内存中. 然后再对特定的 Server 的配置数据进行访问.
172
173 用法:
174
175 .. code-block:: python
176
177 >>> config_loader = ConfigLoader.new(env_name="sbx")
178 >>> for server_name, server in config_loader.iter_servers():
179 ...
180 >>> server = config_loader.get_server(server_name="blue")
181 >>> server
182 Server(id='sbx-blue', db_admin_password='sbx*dummy4test', db_username='myuser', db_password='sbx*dummy4test')
183 """
184
185 _env: Env = dataclasses.field(init=False) # a cache of the env specific config
186
187 @classmethod
188 def new(
189 cls,
190 env_name: str,
191 parameter_name_prefix: T.Optional[str] = None,
192 s3folder_config: T.Optional[str] = None,
193 bsm: "BotoSesManager" = default_bsm,
194 ) -> "ConfigLoader":
195 """
196 创建一个新的 ConfigLoader 对象,
197
198 :param env_name: the environment name of the env specific config you want to
199 load from, stx, tst, prd, etc. If None, then load the master config.
200 :param parameter_name_prefix: the parameter name prefix, the full name will
201 be ${parameter_name_prefix}-${env_name}.
202 :param s3folder_config: S3 配置数据的根目录, 默认为
203 s3://aws_account_id}-{aws_region}-artifacts/projects/acore_server_config/config/
204 :param bsm: ``boto_session_manager.BotoSesManager`` object, if not provided,
205 then use the current runtime default AWS CLI profile.
206 """
207 config = get_config(
208 bsm=bsm,
209 parameter_name_prefix=parameter_name_prefix,
210 env_name=env_name,
211 s3folder_config=s3folder_config,
212 )
213 env = config.get_env(env_name)
214 config_loader = cls()
215 config_loader._env = env
216 return config_loader
217
218 def iter_servers(self) -> T.Iterable[T.Tuple[str, Server]]:
219 """
220 遍历所有的 server. 返回许多 (server_name, server) 的 tuple. 这类似于字典中的
221 ``dict.items()`` 方法
222 """
223 return self._env.servers.items()
224
225 def get_server(self, server_name: str) -> Server:
226 """
227 获得特定 server 的配置数据.
228
229 :param server_name: 服务器的名字 (不包括环境名, 包括环境名的字符串是 server_id).
230 例如 "blue", "green" 等.
231 """
232 return self._env.servers[server_name]
233# [configloader-end]
Note
TODO: 介绍一下我们的 boostrap 程序是如何用这个库来将配置数据应用到游戏服务器上的.
Deploy Configuration Data#
管理员用 1. Load Configuration Data From Local Files 中的方法从本地配置文件中读取配置数据后, 就可以将其部署到 AWS S3 上了. config/deploy_parameters.py 脚本里有示例代码. 我个人也是用这个脚本来部署配置数据的. 该脚本的源码如下:
config/deploy_parameters.py
1# -*- coding: utf-8 -*-
2
3"""
4将所有服务器的配置数据部署到 AWS S3. 在这个项目中, 因为集群上的服务器数量多, 配置的内容复杂,
5最终配置数据可能会很大. 只有用 AWS S3 才可以存储任意多, 任意大的数据.
6
7- Edit non secret config data: ``~/Documents/GitHub/acore_server_config-project/config/config.json``
8- Edit secret config data: ``~/.projects/acore_server_config/config-secret.json``
9"""
10
11from s3pathlib import S3Path
12from acore_server_config.boto_ses import bsm
13from acore_server_config.config.init import config
14
15s3folder_config = (
16 S3Path(f"s3://{bsm.aws_account_alias}-{bsm.aws_region}-artifacts")
17 .joinpath(
18 "projects",
19 "acore_server_config",
20 "config",
21 )
22 .to_dir()
23)
24config.deploy(bsm=bsm, s3folder_config=s3folder_config)
25# config.delete(bsm=bsm, s3folder_config=s3folder_config, include_history=True)
每次部署的时候, 我们不会 overwrite 已经存在的配置数据, 而是自动创建一个新的版本. 在 AWS S3 上的目录结构如下:
# 配置数据的根目录
s3://bucket/projects/acore_server_config/config/
# 这个目录下的配置数据文件是包含了所有环境的配置数据
s3://bucket/projects/acore_server_config/config/acore_server_config/
# 这几个目录只包含了属于自己环境的配置数据
s3://bucket/projects/acore_server_config/config/acore_server_config-sbx/
s3://bucket/projects/acore_server_config/config/acore_server_config-tst/
s3://bucket/projects/acore_server_config/config/acore_server_config-prd/
# 我们就拿 production 为例, 其他几个文件夹下的结构类似
# 每个环境的配置都会有一个 latest 文件和所有的历史版本文件, latest 中的数据永远和最新的历史版本一样
s3://bucket/projects/acore_server_config/config/acore_server_config-prd/acore_server_config-prd-latest.json
s3://bucket/projects/acore_server_config/config/acore_server_config-prd/acore_server_config-prd-000001.json
s3://bucket/projects/acore_server_config/config/acore_server_config-prd/acore_server_config-prd-000002.json
s3://bucket/projects/acore_server_config/config/acore_server_config-prd/acore_server_config-prd-000003.json
Note
之所以不用 parameter store 是因为配置数据可能会很大
I Lost My Local Secret Configuration File#
如果你是管理员, 并且不慎将本地的敏感配置数据文件删除了, 那么你可以到 AWS S3 上去找到 s3://bucket/projects/acore_server_config/config/acore_server_config/acore_server_config-latest.json
文件将其下载到本地, 然后把 secret_data
key 下面的内容复制到 ${HOME}/.projects/acore_server_config/config-secret.json
文件中既可.
Update Game Server Config By Restarting EC2#
当你要更改游戏服务器配置时, 你通常需要将游戏服务器临时关闭, 更新配置, 然后重新启动. 由于我们的 EC2 重启时有 bootstrap 程序, 会从 AWS S3 从新读取并应用配置数据. 所以你可以简单的关闭 world server, 然后关闭 EC2, 用 Deploy Configuration Data 中的方法更新数据, 然后启动 EC2 既可.
Update Game Server Config Without Restarting EC2#
这一节介绍了如何在只重启 world server, 但是不重启 EC2 的情况下更新配置数据.
TODO 以后再补充说如何做.