• R/O
  • HTTP
  • SSH
  • HTTPS

grid-chef-repo: コミット

Grid環境構築用のChefリポジトリです。


コミットメタ情報

リビジョンd707e21cebee48f18feb35e24b56e3868e011337 (tree)
日時2018-02-07 20:33:13
作者whitestar <whitestar@user...>
コミッターwhitestar

ログメッセージ

adds Minio support.

変更サマリ

差分

--- a/cookbooks/screwdriver/CHANGELOG.md
+++ b/cookbooks/screwdriver/CHANGELOG.md
@@ -1,5 +1,9 @@
11 # screwdriver CHANGELOG
22
3+0.6.0
4+-----
5+- adds Minio support.
6+
37 0.5.0
48 -----
59 - adds PostgreSQL support.
--- a/cookbooks/screwdriver/README.md
+++ b/cookbooks/screwdriver/README.md
@@ -5,6 +5,7 @@ This cookbook sets up a Screwdriver CI/CD service by Docker Compose.
55
66 ## Contents
77
8+- [Contents](#contents)
89 - [Requirements](#requirements)
910 - [platforms](#platforms)
1011 - [packages](#packages)
@@ -22,6 +23,7 @@ This cookbook sets up a Screwdriver CI/CD service by Docker Compose.
2223 - [Database username management (for MySQL, PostgreSQL,...) by Chef Vault](#database-username-management-for-mysql-postgresql-by-chef-vault)
2324 - [Database password management (for MySQL, PostgreSQL,...) by Chef Vault](#database-password-management-for-mysql-postgresql-by-chef-vault)
2425 - [Database root password management (for MySQL, PostgreSQL,...) by Chef Vault](#database-root-password-management-for-mysql-postgresql-by-chef-vault)
26+ - [S3 (compatible) server access key management by Chef Vault](#s3-compatible-server-access-key-management-by-chef-vault)
2527 - [OAuth client ID, secret and GitHub webhook secret management by Chef Vault](#oauth-client-id-secret-and-github-webhook-secret-management-by-chef-vault)
2628 - [Note](#note)
2729 - [Database Initialization](#database-initialization)
@@ -55,9 +57,12 @@ This cookbook sets up a Screwdriver CI/CD service by Docker Compose.
5557 |`['screwdriver']['db_username_vault_item']`|Hash|Optional, Sets a database username from Chef Vault. See `attributes/default.rb`|`{}`|
5658 |`['screwdriver']['db_password_vault_item']`|Hash|Optional, Sets a database password from Chef Vault. See `attributes/default.rb`|`{}`|
5759 |`['screwdriver']['db_root_password_vault_item']`|Hash|Optional, Sets a database password for the root user from Chef Vault. See `attributes/default.rb`|`{}`|
60+|`['screwdriver']['s3_access_key_id_vault_item']`|Hash|Optional, Sets a S3 access key id from Chef Vault. See `attributes/default.rb`|`{}`|
61+|`['screwdriver']['s3_access_key_secret_vault_item']`|Hash|Optional, Sets a S3 access key secret from Chef Vault. See `attributes/default.rb`|`{}`|
5862 |`['screwdriver']['ui']['tls_setup_mode']`|String|`'reverseproxy'` only. Note: [_Add TLS support to UI docker container #377_](https://github.com/screwdriver-cd/screwdriver/issues/377)|`'reverseproxy'`|
5963 |`['screwdriver']['api']['config']`|Hash|This hash object is expanded to a `/config/local.yaml` file in the API Docker container.|See `attributes/default.rb`|
6064 |`['screwdriver']['api']['scms_vault_items']`|Hash|This hash contains Chef Vault item definitions of SCM's secrets.|See `attributes/default.rb`|
65+|`['screwdriver']['store']['backend']`|String|`nil` (in memory) or `'minio'`.|`nil`|
6166 |`['screwdriver']['store']['config']`|Hash|This hash object is expanded to a `/config/local.yaml` file in the Store Docker container.|See `attributes/default.rb`|
6267 |`['screwdriver']['docker-compose']['import_ca']`|Boolean|whether import internal CA certificates or not.|`false`|
6368 |`['screwdriver']['docker-compose']['app_dir']`|String|Path string.|`"#{node['docker-grid']['compose']['app_dir']}/screwdriver"`|
@@ -589,6 +594,50 @@ override_attributes(
589594 )
590595 ```
591596
597+### S3 (compatible) server access key management by Chef Vault
598+
599+- create vault items.
600+
601+```text
602+$ cat ~/sec/tmp/screwdriver_s3_access_key.json
603+{
604+ "kid":"********************",
605+ "secret":"****************************************"
606+}
607+
608+$ cd $CHEF_REPO_PATH
609+$ knife vault create screwdriver s3_access_key --json ~/sec/tmp/screwdriver_s3_access_key.json
610+```
611+
612+- grant reference permission to the screwdriver host
613+
614+```text
615+$ knife vault update screwdriver s3_access_key -S 'name:screwdriver-host.example.com'
616+```
617+
618+- modify attributes
619+
620+```ruby
621+override_attributes(
622+ 'screwdriver' => {
623+ # ...
624+ 's3_access_key_id_vault_item' => {
625+ 'vault' => 'screwdriver',
626+ 'name' => 's3_access_key',
627+ 'env_context' => false,
628+ 'key' => 'kid',
629+ },
630+ 's3_access_key_secret_vault_item' => {
631+ 'vault' => 'screwdriver',
632+ 'name' => 's3_access_key',
633+ 'env_context' => false,
634+ 'key' => 'secret',
635+ },
636+ # ...
637+ },
638+)
639+```
640+
592641 ### OAuth client ID, secret and GitHub webhook secret management by Chef Vault
593642
594643 - create vault items.
--- a/cookbooks/screwdriver/attributes/default.rb
+++ b/cookbooks/screwdriver/attributes/default.rb
@@ -30,10 +30,10 @@ default['screwdriver']['jwt_private_key_vault_item'] = {
3030 =begin
3131 'vault' => 'screwdriver',
3232 'name' => 'jwt_private_key',
33- # single password or nested hash password path delimited by slash
33+ # single secret or nested hash secret path delimited by slash
3434 'env_context' => false,
35- 'key' => 'private', # real hash path: "/password"
36- # or nested hash password path delimited by slash
35+ 'key' => 'private', # real hash path: "/private"
36+ # or nested hash secret path delimited by slash
3737 #'env_context' => true,
3838 #'key' => 'hash/path/to/private', # real hash path: "/#{node.chef_environment}/hash/path/to/private"
3939 =end
@@ -42,10 +42,10 @@ default['screwdriver']['jwt_public_key_vault_item'] = {
4242 =begin
4343 'vault' => 'screwdriver',
4444 'name' => 'jwt_public_key',
45- # single password or nested hash password path delimited by slash
45+ # single secret or nested hash secret path delimited by slash
4646 'env_context' => false,
47- 'key' => 'public', # real hash path: "/password"
48- # or nested hash password path delimited by slash
47+ 'key' => 'public', # real hash path: "/public"
48+ # or nested hash secret path delimited by slash
4949 #'env_context' => true,
5050 #'key' => 'hash/path/to/public', # real hash path: "/#{node.chef_environment}/hash/path/to/public"
5151 =end
@@ -83,7 +83,7 @@ default['screwdriver']['db_username_vault_item'] = {
8383 # single usernaem or nested hash username path delimited by slash
8484 'env_context' => false,
8585 'key' => 'username', # real hash path: "/username"
86- # or nested hash password path delimited by slash
86+ # or nested hash username path delimited by slash
8787 #'env_context' => true,
8888 #'key' => 'hash/path/to/username', # real hash path: "/#{node.chef_environment}/hash/path/to/username"
8989 =end
@@ -112,6 +112,30 @@ default['screwdriver']['db_root_password_vault_item'] = {
112112 #'key' => 'hash/path/to/password', # real hash path: "/#{node.chef_environment}/hash/path/to/password"
113113 =end
114114 }
115+default['screwdriver']['s3_access_key_id_vault_item'] = {
116+=begin
117+ 'vault' => 'screwdriver',
118+ 'name' => 's3_access_key',
119+ # single key id or nested hash key id path delimited by slash
120+ 'env_context' => false,
121+ 'key' => 'kid', # real hash path: "/kid"
122+ # or nested hash key id path delimited by slash
123+ #'env_context' => true,
124+ #'key' => 'hash/path/to/kid', # real hash path: "/#{node.chef_environment}/hash/path/to/kid"
125+=end
126+}
127+default['screwdriver']['s3_access_key_secret_vault_item'] = {
128+=begin
129+ 'vault' => 'screwdriver',
130+ 'name' => 's3_access_key',
131+ # single secret or nested hash secret path delimited by slash
132+ 'env_context' => false,
133+ 'key' => 'secret', # real hash path: "/secret"
134+ # or nested hash secret path delimited by slash
135+ #'env_context' => true,
136+ #'key' => 'hash/path/to/secret', # real hash path: "/#{node.chef_environment}/hash/path/to/secret"
137+=end
138+}
115139
116140 force_override['screwdriver']['ui']['tls_setup_mode'] = 'reverseproxy'
117141 # These hash objects are expanded to a `/config/local.yaml` file in each Docker container.
@@ -201,11 +225,26 @@ default['screwdriver']['api']['scms_vault_items'] = {
201225 =end
202226 }
203227
228+default['screwdriver']['store']['backend'] = nil # or 'minio'
204229 default['screwdriver']['store']['config'] = {
205230 'auth' => {},
206231 'httpd' => {
207232 'tls' => false,
208233 },
234+=begin
235+ # for Minio
236+ 'strategy' => {
237+ 'plugin' => 's3',
238+ 's3' => {
239+ 'accessKeyId' => '',
240+ 'secretAccessKey' => '****************************************',
241+ 'region' => 'us-east-1',
242+ 'bucket' => 'screwdriver',
243+ 'endpoint' => 'http://s3:9000/screwdriver',
244+ 'signatureVersion' => 'v4',
245+ },
246+ },
247+=end
209248 }
210249
211250 # Useless?!
@@ -391,12 +430,34 @@ EOS
391430 'PORT' => '80',
392431 'URI' => "http://#{cn}:9002",
393432 #'URI' => "http://#{node['ipaddress']}:9002", # unrecommended
394- #'STRATEGY' => 'memory',
395- # This variable will be set by the screwdriver::docker-compose recipe automatically.
433+ # These variables will be set by the screwdriver::docker-compose recipe automatically.
396434 #'ECOSYSTEM_UI' => "http://#{cn}:9000", # Better
397435 #'ECOSYSTEM_UI' => "http://#{node['ipaddress']}:9000",
398436 #'ECOSYSTEM_UI' => 'http://ui', # NG for an access from a client.
437+ #'STRATEGY' => 'memory', # default
438+ # * AWS S3
439+ #'STRATEGY' => 's3',
440+ # If node['screwdriver']['s3_access_key_{id,secret}_vault_item'] is set,
441+ # these 2 variables will be set by the screwdriver::docker-compose recipe automatically.
442+ #'S3_ACCESS_KEY_ID' => '${S3_ACCESS_KEY_ID}',
443+ #'S3_ACCESS_KEY_SECRET' => '${S3_ACCESS_KEY_SECRET}',
444+ #'S3_REGION' => 'us-east-1',
445+ #'S3_BUCKET' => 'screwdriver',
446+ # * Minio
447+ # If node['screwdriver']['store']['backend'] is 'minio',
448+ # these variables will be set by the screwdriver::docker-compose recipe automatically.
449+ #'STRATEGY' => 's3',
450+ #'S3_ACCESS_KEY_ID' => '${S3_ACCESS_KEY_ID}',
451+ #'S3_ACCESS_KEY_SECRET' => '${S3_ACCESS_KEY_SECRET}',
452+ #'S3_REGION' => 'us-east-1',
453+ #'S3_BUCKET' => 'screwdriver',
454+ #'S3_ENDPOINT' => 'http://s3:9000/screwdriver', # tricky!! setting for the S3 virtual hosting style.
455+ #'S3_SIG_VER' => 'v4',
399456 },
457+ # for S3 compatible server
458+ #'links' => [
459+ # 'screwdriver.s3',
460+ #],
400461 },
401462 },
402463 }
@@ -435,4 +496,25 @@ when 'postgres'
435496 }
436497 end
437498
499+# S3 compatible server
500+case node['screwdriver']['store']['backend']
501+when 'minio'
502+ version_2_config['services']['screwdriver.s3'] = {
503+ 'image' => 'minio/minio',
504+ 'ports' => [
505+ #'9010:9000', # default
506+ ],
507+ 'command' => 'server /export',
508+ 'volumes' => [
509+ # This variable will be set by the screwdriver::docker-compose recipe automatically.
510+ #"#{node['screwdriver']['docker-compose']['data_dir']}//minio:/export:rw",
511+ ],
512+ 'environment' => {
513+ # These variables will be set by the screwdriver::docker-compose recipe automatically.
514+ #'MINIO_ACCESS_KEY' => '${S3_ACCESS_KEY_ID}',
515+ #'MINIO_SECRET_KEY' => '${S3_ACCESS_KEY_SECRET}',
516+ },
517+ }
518+end
519+
438520 default['screwdriver']['docker-compose']['config'] = version_2_config
--- a/cookbooks/screwdriver/recipes/docker-compose.rb
+++ b/cookbooks/screwdriver/recipes/docker-compose.rb
@@ -259,17 +259,17 @@ if db_dialect != 'sqlite'
259259 when 'mysql'
260260 mysql_data_dir = "#{data_dir}/mysql"
261261 resources(directory: mysql_data_dir) rescue directory mysql_data_dir do
262- owner 'root'
263- group 'root'
262+ owner 999
263+ group 'docker'
264264 mode '0755'
265265 recursive true
266266 end
267267
268+ db_vols.push("#{mysql_data_dir}:/var/lib/mysql:rw")
268269 db_envs['MYSQL_DATABASE'] = api_envs_org['DATASTORE_SEQUELIZE_DATABASE']
269270 db_envs['MYSQL_USER'] = '${DB_USERNAME}' unless db_username.nil?
270271 db_envs['MYSQL_PASSWORD'] = '${DB_PASSWORD}' unless db_password.nil?
271272 db_envs['MYSQL_ROOT_PASSWORD'] = '${DB_ROOT_PASSWORD}' unless db_root_password.nil?
272- db_vols.push("#{mysql_data_dir}:/var/lib/mysql:rw")
273273 when 'postgres'
274274 pg_data_dir = "#{data_dir}/postgres"
275275 resources(directory: pg_data_dir) rescue directory pg_data_dir do
@@ -279,11 +279,11 @@ if db_dialect != 'sqlite'
279279 recursive true
280280 end
281281
282+ db_vols.push("#{pg_data_dir}:/database:rw")
282283 db_envs['POSTGRES_DB'] = api_envs_org['DATASTORE_SEQUELIZE_DATABASE']
283284 db_envs['POSTGRES_USER'] = '${DB_USERNAME}' unless db_username.nil?
284285 db_envs['POSTGRES_PASSWORD'] = '${DB_PASSWORD}' unless db_password.nil?
285286 db_envs['PGDATA'] = '/database'
286- db_vols.push("#{pg_data_dir}:/database:rw")
287287 end
288288 end
289289
@@ -305,6 +305,7 @@ else
305305 end
306306
307307 # store
308+store_backend = node['screwdriver']['store']['backend']
308309 store_envs_org = config_srvs['store']['environment']
309310 store_envs = {}
310311 store_vols = config_srvs['store']['volumes'].to_a
@@ -321,6 +322,63 @@ else
321322 }
322323 end
323324
325+s3_access_key_id = nil
326+s3_access_key_id_vault_item = node['screwdriver']['s3_access_key_id_vault_item']
327+unless s3_access_key_id_vault_item.empty?
328+ s3_access_key_id = get_vault_item_value(s3_access_key_id_vault_item)
329+ store_envs['S3_ACCESS_KEY_ID'] = '${S3_ACCESS_KEY_ID}'
330+end
331+
332+s3_access_key_secret = nil
333+s3_access_key_secret_vault_item = node['screwdriver']['s3_access_key_secret_vault_item']
334+unless s3_access_key_secret_vault_item.empty?
335+ s3_access_key_secret = get_vault_item_value(s3_access_key_secret_vault_item)
336+ store_envs['S3_ACCESS_KEY_SECRET'] = '${S3_ACCESS_KEY_SECRET}'
337+end
338+
339+# S3 compatible server
340+if !store_backend.nil? && !store_backend.empty?
341+ override_config_srvs['store']['links'] = ['screwdriver.s3']
342+ store_envs['STRATEGY'] = 's3'
343+ store_envs['S3_BUCKET'] = 'screwdriver'
344+
345+ #s3_envs_org = config_srvs['screwdriver.s3']['environment']
346+ s3_envs = {}
347+ s3_vols = config_srvs['screwdriver.s3']['volumes'].to_a
348+
349+ s3_port = '9010' # default
350+ s3_in_port = '9000'
351+ ports = config_srvs['screwdriver.s3']['ports']
352+
353+ case store_backend
354+ when 'minio'
355+ store_envs['S3_REGION'] = 'us-east-1'
356+ store_envs['S3_ENDPOINT'] = "http://s3:#{s3_in_port}/screwdriver" # for path style
357+ store_envs['S3_SIG_VER'] = 'v4'
358+
359+ if ports.empty?
360+ override_config_srvs['screwdriver.s3']['ports'] = ["#{s3_port}:#{s3_in_port}"]
361+ else
362+ ports.each {|port|
363+ elms = port.split(':')
364+ s3_port = (elms.size == 2 ? elms[0] : elms[1]) if elms.last == s3_in_port
365+ }
366+ end
367+
368+ minio_data_dir = "#{data_dir}/minio"
369+ resources(directory: minio_data_dir) rescue directory minio_data_dir do
370+ owner 'root'
371+ group 'root'
372+ mode '0755'
373+ recursive true
374+ end
375+
376+ s3_vols.push("#{minio_data_dir}:/export:rw")
377+ s3_envs['MINIO_ACCESS_KEY'] = '${S3_ACCESS_KEY_ID}' unless s3_access_key_id.nil?
378+ s3_envs['MINIO_SECRET_KEY'] = '${S3_ACCESS_KEY_SECRET}' unless s3_access_key_secret.nil?
379+ end
380+end
381+
324382 override_store_config['auth']['jwtPublicKey'] = jwt_public_key
325383 # Note: prevent Chef from logging JWT key attribute value. (=> template variables)
326384 # However Docker env file format does not support multi-line value and backslash escaped string yet.
@@ -467,6 +525,9 @@ force_override_config_srvs['store']['environment'] = store_envs unless store_env
467525 if db_dialect != 'sqlite'
468526 force_override_config_srvs['db']['environment'] = db_envs unless db_envs.empty?
469527 end
528+if !store_backend.nil? && !store_backend.empty?
529+ force_override_config_srvs['screwdriver.s3']['environment'] = s3_envs unless s3_envs.empty?
530+end
470531 # reset vlumes array.
471532 override_config_srvs['api']['volumes'] = api_vols unless api_vols.empty?
472533 override_config_srvs['ui']['volumes'] = ui_vols unless ui_vols.empty?
@@ -474,6 +535,9 @@ override_config_srvs['store']['volumes'] = store_vols unless store_vols.empty?
474535 if db_dialect != 'sqlite'
475536 override_config_srvs['db']['volumes'] = db_vols unless db_vols.empty?
476537 end
538+if !store_backend.nil? && !store_backend.empty?
539+ override_config_srvs['screwdriver.s3']['volumes'] = s3_vols unless s3_vols.empty?
540+end
477541
478542 template env_file do
479543 source 'opt/docker-compose/app/screwdriver/.env'
@@ -489,6 +553,8 @@ template env_file do
489553 db_username: db_username,
490554 db_password: db_password,
491555 db_root_password: db_root_password,
556+ s3_access_key_id: s3_access_key_id,
557+ s3_access_key_secret: s3_access_key_secret,
492558 # **DEPRECATED!!**
493559 # JWT keys setting -> /config/local.yaml
494560 #jwt_private_key: jwt_private_key,
--- a/cookbooks/screwdriver/templates/default/opt/docker-compose/app/screwdriver/.env
+++ b/cookbooks/screwdriver/templates/default/opt/docker-compose/app/screwdriver/.env
@@ -33,3 +33,9 @@ DB_PASSWORD=<%= @db_password %>
3333 <% unless @db_root_password.nil? %>
3434 DB_ROOT_PASSWORD=<%= @db_root_password %>
3535 <% end %>
36+<% unless @s3_access_key_id.nil? %>
37+S3_ACCESS_KEY_ID=<%= @s3_access_key_id %>
38+<% end %>
39+<% unless @s3_access_key_secret.nil? %>
40+S3_ACCESS_KEY_SECRET=<%= @s3_access_key_secret %>
41+<% end %>
--- a/cookbooks/screwdriver/version
+++ b/cookbooks/screwdriver/version
@@ -1 +1 @@
1-0.5.0
1+0.6.0
旧リポジトリブラウザで表示