Skip to content

Commit 9b25c8c

Browse files
vsochvsoch
committed
refactor to use a single settings.py, allow a settings.yaml
Signed-off-by: vsoch <vsoch@users.noreply.github.com>
1 parent 404f138 commit 9b25c8c

17 files changed

Lines changed: 608 additions & 655 deletions

File tree

‎docs/_docs/install/settings.md‎

Lines changed: 50 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,16 @@ toc: false
66

77
# Settings
88

9-
See that folder called [settings](https://github.com/singularityhub/sregistry/blob/master/shub/settings)? inside are a bunch of different starting settings for the application. We will change them in these files before we start the application. There are actually only two files you need to poke into, generating a `settings/secrets.py` from our template [settings/dummy_secrets.py](https://github.com/singularityhub/sregistry/blob/master/shub/settings/dummy_secrets.py) for application secrets, and [settings/config.py](https://github.com/singularityhub/sregistry/blob/master/shub/settings/config.py) to configure your database and registry information.
9+
As of version `1.1.41` we have one core [settings.py](https://github.com/singularityhub/sregistry/blob/master/shub/settings.py) file
10+
that can be populated in several ways:
11+
12+
- Directly in the file (e.g., if you build your own base container)
13+
- From the environment (described below with a `SREGISTRY_` prefix
14+
- From a config file, `settings.yaml` in the root directory of your server OR custom set at `SREGISTRY_SETTINGS_FILE`
15+
16+
You will want to choose a strategy that works for your deployment, and then tweak the values to your liking before we start the application.
17+
For example, if you are running sregistry on a filesystem directly that you can access, you can easily write secrets into a `secrets.py` in the
18+
`shub/` directory alongside settings and you can use the [dummy_secrets.py](https://github.com/singularityhub/sregistry/blob/master/shub/dummy_secrets.py) as a starter template. However, if you are deploying via Kubernetes or similar, you can either choose to define secrets entirely in the environment, or you can have a `settings.yaml` that is created in a manifest and passed to the application.
1019

1120
## Environment
1221

@@ -29,8 +38,24 @@ are exposed (all strings):
2938
- SREGISTRY_DATABASE_HOST (db)
3039
- SREGISTRY_DATABASE_PORT (5432)
3140

32-
And you can set a custom set of comma separated values for either of `SREGISTRY_ADMINS` or `SREGISTRY_PLUGINS_ENABLED`.
33-
Each of these settings is explained in more detail in the sections below.
41+
And you can set a custom set of comma separated values (in a string) for either of `SREGISTRY_ADMINS` or `SREGISTRY_PLUGINS_ENABLED`.
42+
43+
```console
44+
SREGISTRY_ADMINS="adminA,adminB,adminC"
45+
SREGISTRY_PLUGINS_ENABLED="ldap,google-build"
46+
```
47+
48+
Finally, if you need to set secrets from the environment (and don't want to provide the secrets.py file) you can
49+
do the same by taking any secret value and prefixing it with `SREGISTRY_SECRET_`. For this approach, we currently
50+
support strings and booleans (true/false derivatives) and we can add support for other types if requested.
51+
52+
For example, to set your secret key (required) you would do:
53+
54+
```console
55+
SREGISTRY_SECRET_SECRET_KEY = 'xxxxxxxxxxxxxxxxxx'
56+
```
57+
For any setting in the sections below, you can set them in the environment (with `SREGISTRY_` or `SREGISTRY_SECRET` prefix for secrets)
58+
or via the settings.yaml file. Each of these settings is explained in more detail in the sections below.
3459

3560
## Secrets
3661

@@ -39,13 +64,13 @@ There should be a file called `secrets.py` in the shub settings folder (it won't
3964
An template to work from is provided in the settings folder called `dummy_secrets.py`. You can copy this template:
4065

4166
```bash
42-
cp shub/settings/dummy_secrets.py shub/settings/secrets.py
67+
cp shub/dummy_secrets.py shub/secrets.py
4368
```
4469

4570
Or, if you prefer a clean secrets file, create a blank one as below:
4671

4772
```bash
48-
touch shub/settings/secrets.py
73+
touch shub/secrets.py
4974
```
5075

5176
and inside you want to add a `SECRET_KEY`. You can use the [secret key generator](http://www.miniwebtool.com/django-secret-key-generator/) to make a new secret key, and call it `SECRET_KEY` in your `secrets.py` file, like this:
@@ -54,9 +79,15 @@ and inside you want to add a `SECRET_KEY`. You can use the [secret key generator
5479
SECRET_KEY = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
5580
```
5681

82+
And you can also set this in the environment:
83+
84+
```bash
85+
export SREGISTRY_SECRET_SECRET_KEY=xxxxxxxxxxxxxxxxxx
86+
```
87+
5788
### Authentication Secrets
5889

59-
One thing I (@vsoch) can't do for you in advance is produce application keys and secrets to give your Registry for each social provider that you want to allow users (and yourself) to login with. We are going to use a framework called [python social auth](https://python-social-auth-docs.readthedocs.io/en/latest/configuration/django.html) to achieve this, and in fact you can add a [number of providers](http://python-social-auth-docs.readthedocs.io/en/latest/backends/index.html) (I have set up a lot of them, including SAML, so please <a href="https://www.github.com/singularityhub/sregistry/isses" target="_blank">submit an issue</a> if you want one added to the base proper.). Singularity Registry uses OAuth2 with a token--> refresh flow because it gives the user power to revoke permission at any point, and is a more modern strategy than storing a database of usernames and passwords. You can enable or disable as many of these that you want, and this is done in the [settings/config.py](https://github.com/singularityhub/sregistry/blob/master/shub/settings/config.py):
90+
One thing I (@vsoch) can't do for you in advance is produce application keys and secrets to give your Registry for each social provider that you want to allow users (and yourself) to login with. We are going to use a framework called [python social auth](https://python-social-auth-docs.readthedocs.io/en/latest/configuration/django.html) to achieve this, and in fact you can add a [number of providers](http://python-social-auth-docs.readthedocs.io/en/latest/backends/index.html) (I have set up a lot of them, including SAML, so please <a href="https://www.github.com/singularityhub/sregistry/isses" target="_blank">submit an issue</a> if you want one added to the base proper.). Singularity Registry uses OAuth2 with a token--> refresh flow because it gives the user power to revoke permission at any point, and is a more modern strategy than storing a database of usernames and passwords. You can enable or disable as many of these that you want, and this is done in the [settings.py](https://github.com/singularityhub/sregistry/blob/master/shub/settings.py), which is controllable via the environment or a `settings.yaml` or by editing the file directly:
6091

6192

6293
```python
@@ -69,7 +100,8 @@ ENABLE_BITBUCKET_AUTH=False
69100
ENABLE_GITHUB_ENTERPRISE_AUTH=False
70101
```
71102

72-
and you will need at least one to log in. I've found that GitHub works the fastest and easiest, and then Google.
103+
Any of the above can be set in the environment with the `SREGISTRY_` prefix.
104+
You will need at least one to log in. I've found that GitHub works the fastest and easiest, and then Google.
73105
Twitter now requires an actual server name and won't work with localhost, but if you are deploying on a server with a proper domain go ahead and use it. All avenues are extremely specific with regard to callback urls, so you should be very careful in setting them up. If you want automated builds from a repository
74106
integration with Google Cloud Build, then you must use GitHub.
75107

@@ -158,7 +190,6 @@ Where the key and secret are replaced by the ones given to you. If you have a pr
158190
SOCIAL_AUTH_GITLAB_API_URL = 'https://example.com'
159191
```
160192

161-
162193
### Bitbucket OAuth2
163194

164195
We will be using the [bitbucket](https://python-social-auth.readthedocs.io/en/latest/backends/bitbucket.html) backend for Python Social Auth.
@@ -188,7 +219,7 @@ SOCIAL_AUTH_BITBUCKET_OAUTH2_SECRET = '<your-consumer-secret>'
188219
SOCIAL_AUTH_BITBUCKET_OAUTH2_VERIFIED_EMAILS_ONLY = True
189220
```
190221

191-
Finally, don't forget to enable the bitbucket login in settings/config.py:
222+
Finally, don't forget to enable the bitbucket login in settings.py:
192223

193224
```python
194225
ENABLE_BITBUCKET_AUTH=True
@@ -206,24 +237,19 @@ Note that Twitter now does not accept localhost urls. Thus,
206237
the callback url here should be `http://[your-domain]/complete/twitter`.
207238

208239

209-
210-
## Config
211-
212-
In the [config.py](https://github.com/singularityhub/sregistry/blob/master/shub/settings/config.py) you need to define the following:
213-
214240
### Google Analytics
215241

216-
If you want to add a Google analytics code, you can do this in the settings/config.py:
242+
If you want to add a Google analytics code, you can do this in the settings.py:
217243

218244
```python
219245
GOOGLE_ANALYTICS = "UA-XXXXXXXXX"
220246
```
221247

222248
The default is set to None, and doesn't add analytics to the registry.
223249

224-
225250
### Domain Name
226-
A Singularity Registry Server should have a domain. It's not required, but it makes it much easier for yourself and users to remember the address. The first thing you should do is change the `DOMAIN_NAME_*` variables in your settings [settings/config.py](https://github.com/singularityhub/sregistry/blob/master/shub/settings/config.py#L30).
251+
252+
A Singularity Registry Server should have a domain. It's not required, but it makes it much easier for yourself and users to remember the address. The first thing you should do is change the `DOMAIN_NAME_*` variables in your settings [settings.py](https://github.com/singularityhub/sregistry/blob/master/shub/settings.py).
227253

228254
For local testing, you will want to change `DOMAIN_NAME` and `DOMAIN_NAME_HTTP` to be localhost. Also note that I've set the regular domain name (which should be https) to just http because I don't have https locally:
229255

@@ -238,6 +264,7 @@ It's up to the deployer to set one up a domain or subdomain for the server. Typi
238264

239265

240266
### Registry Contact
267+
241268
You need to define a registry uri, and different contact information:
242269

243270
```python
@@ -365,7 +392,7 @@ For other disable and limit arguments (for GitHub, creating, or receiving builds
365392

366393
By default, the database itself will be deployed as a postgres image called `db`. You probably don't want this for production (for example, I use a second instance with postgres and a third with a hot backup, but it's an ok solution for a small cluster or single user. Either way, we recommend backing it up every so often.
367394

368-
When your database is set up, you can define it in your `secrets.py` and it will override the Docker image one in the `settings/main.py file`. It should look something like this
395+
When your database is set up, you can define it in your `secrets.py` and it will override the Docker image one in the `settings.py file`. It should look something like this
369396

370397
```python
371398
DATABASES = {
@@ -381,6 +408,7 @@ DATABASES = {
381408
```
382409

383410
### Logging
411+
384412
By default, Singularity Registry keeps track of all requests to pull containers, and you have control over the level of detail that is kept. If you want to save complete metadata (meaning the full json response for each call) then you should set `LOGGING_SAVE_RESPONSES` to True. If you expect heavy use and want to save the minimal (keep track of which collections are pulled how many times) the reccomendation is to set this to False.
385413

386414
```python
@@ -389,28 +417,18 @@ LOGGING_SAVE_RESPONSES=True
389417

390418
## API
391419

392-
Take a look in [settings/api.py](https://github.com/singularityhub/sregistry/blob/master/shub/settings/api.py)
393-
to configure your restful API. You can uncomment the first block to require authentication:
420+
To configure your restful API you can set `SREGISTRY_API_REQUIRE_AUTH` to true.
394421

395422
```python
396423
'DEFAULT_PERMISSION_CLASSES': (
397424
'rest_framework.permissions.IsAuthenticated',
398425
),
399426
```
400427

401-
And also choose throttle rates for users and anonymous API requests.
428+
And also choose throttle rates for users and anonymous API requests
429+
with the following:
430+
402431

403-
```python
404-
# You can also customize the throttle rates, for anon and users
405-
'DEFAULT_THROTTLE_CLASSES': (
406-
'rest_framework.throttling.AnonRateThrottle',
407-
),
408-
# https://www.django-rest-framework.org/api-guide/throttling/
409-
'DEFAULT_THROTTLE_RATES': {
410-
'anon': '100/day',
411-
'user': '1000/day',
412-
},
413-
```
414432

415433
These are important metrics to ensure that your server isn't subject to a DoS attack.
416434

‎requirements.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,3 +42,4 @@ social-auth-core[saml]
4242
sregistry[all-basic]>=0.2.19
4343
uwsgi
4444
minio==5.0.8
45+
pyyaml

shub/settings/dummy_secrets.py renamed to shub/dummy_secrets.py

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
# You must uncomment, and set SECRET_KEY to a secure random value
77
# e.g. https://djskgen.herokuapp.com/
88

9-
# SECRET_KEY = 'xxxxxxxxxxxxxxxxxx'
9+
SECRET_KEY = "xxxxxxxxxxxxxxxxxx"
1010

1111

1212
# =============================================================================
@@ -77,15 +77,15 @@
7777
# GOOGLE_APPLICATION_CREDENTIALS="/path/to/credentials.json"
7878
# SREGISTRY_GOOGLE_PROJECT="myproject-ftw"
7979

80-
# SREGISTRY_GOOGLE_BUILD_CACHE="true"
80+
# GOOGLE_BUILD_CACHE="true"
8181
# After build, do not delete intermediate dependencies in cloudbuild bucket (keep them as cache for rebuild if needed).
8282
# Defaults to being unset, meaning that files are cleaned up. If you define this as anything, the build files will be cached.
8383

84-
# SREGISTRY_GOOGLE_BUILD_LIMIT=100
84+
# GOOGLE_BUILD_LIMIT=100
8585
# To prevent denial of service attacks on Google Cloud Storage, you should set a reasonable limit for the number of active, concurrent builds.
8686
# This number should be based on your expected number of users, repositories, and recipes per repository.
8787

88-
# SREGISTRY_GOOGLE_BUILD_SINGULARITY_VERSION="v3.2.1-slim"
88+
# GOOGLE_BUILD_SINGULARITY_VERSION="v3.2.1-slim"
8989
# if you want to specify a version of Singularity. The version must coincide with a container tag hosted under singularityware/singularity. The version will default to 3.2.0-slim If you want to use a different version, update this variable.
9090

9191
# SREGISTRY_GOOGLE_STORAGE_BUCKET="taco-singularity-registry"
@@ -94,11 +94,11 @@
9494
# Additionally, a temporary bucket is created with the same name ending in _cloudbuild. This bucket is for build time dependencies, and is cleaned up after the fact. If you are having trouble getting a bucket it is likely because the name is taken,
9595
# and we recommend creating both <name> and <name>_cloudbuild in the console and then setting the name here.
9696

97-
# SREGISTRY_GOOGLE_BUILD_TIMEOUT_SECONDS=None
97+
# GOOGLE_BUILD_TIMEOUT_SECONDS=None
9898
# The number of seconds for the build to timeout. If set to None, will be 10 minutes. If
99-
# unset, will default to 3 hours. This time should be less than the SREGISTRY_GOOGLE_BUILD_EXPIRE_SECONDS
99+
# unset, will default to 3 hours. This time should be less than the GOOGLE_BUILD_EXPIRE_SECONDS
100100

101-
# SREGISTRY_GOOGLE_BUILD_EXPIRE_SECONDS=28800
101+
# GOOGLE_BUILD_EXPIRE_SECONDS=28800
102102
# The number of seconds for the build to expire, meaning it's response is no longer accepted by the server. This must be defined.
103103
# The default 28800 indicates 8 hours (in seconds)
104104

‎shub/plugins/google_build/__init__.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@
55
# Ensure that application credentials exist
66
for required in [
77
"GOOGLE_APPLICATION_CREDENTIALS",
8-
"SREGISTRY_GOOGLE_PROJECT",
9-
"SREGISTRY_GOOGLE_BUILD_EXPIRE_SECONDS",
8+
"GOOGLE_PROJECT",
9+
"GOOGLE_BUILD_EXPIRE_SECONDS",
1010
]:
1111
if not hasattr(settings, required):
1212
bot.exit("%s not defined in secrets." % required)

‎shub/plugins/google_build/actions.py‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -209,8 +209,8 @@ def get_build_context():
209209

210210
# Provide all envars directly to client instead of environment
211211
context = {
212-
"GOOGLE_APPLICATION_CREDENTIALS": settings.GOOGLE_APPLICATION_CREDENTIALS,
213-
"SREGISTRY_GOOGLE_PROJECT": settings.SREGISTRY_GOOGLE_PROJECT,
212+
"SREGISTRY_GOOGLE_APPLICATION_CREDENTIALS": settings.GOOGLE_APPLICATION_CREDENTIALS,
213+
"SREGISTRY_GOOGLE_PROJECT": settings.GOOGLE_PROJECT,
214214
}
215215

216216
# Put the credentials in the environment to find
@@ -220,12 +220,12 @@ def get_build_context():
220220

221221
# The following are optional
222222
for attr in [
223-
"SREGISTRY_GOOGLE_BUILD_CACHE",
224-
"SREGISTRY_GOOGLE_BUILD_SINGULARITY_VERSION",
225-
"SREGISTRY_GOOGLE_STORAGE_BUCKET",
223+
"GOOGLE_BUILD_CACHE",
224+
"GOOGLE_BUILD_SINGULARITY_VERSION",
225+
"GOOGLE_STORAGE_BUCKET",
226226
]:
227227
if hasattr(settings, attr):
228-
context[attr] = getattr(settings, attr)
228+
context["SREGISTRY_" + attr] = getattr(settings, attr)
229229
return context
230230

231231

@@ -319,13 +319,13 @@ def is_over_limit(limit=None):
319319
builds. If not set, we use the default in settings.
320320
"""
321321
# Allow the function to set a custom limit
322-
limit = limit or settings.SREGISTRY_GOOGLE_BUILD_LIMIT
322+
limit = limit or settings.GOOGLE_BUILD_LIMIT
323323

324324
# Instantiate client with context (connects to buckets)
325325
context = get_build_context()
326326
client = get_client(debug=True, **context)
327327

328-
project = settings.SREGISTRY_GOOGLE_PROJECT
328+
project = settings.GOOGLE_PROJECT
329329
result = (
330330
client._build_service.projects()
331331
.builds()

‎shub/plugins/google_build/github.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@
2424
DISABLE_BUILDING,
2525
DISABLE_GITHUB,
2626
DOMAIN_NAME,
27-
SREGISTRY_GOOGLE_BUILD_LIMIT,
27+
GOOGLE_BUILD_LIMIT,
2828
)
2929

3030
from .utils import (
@@ -467,7 +467,7 @@ def verify_payload(request, collection):
467467
if is_over_limit():
468468
message = (
469469
"Registry concurrent build limit is "
470-
+ "%s" % SREGISTRY_GOOGLE_BUILD_LIMIT
470+
+ "%s" % GOOGLE_BUILD_LIMIT
471471
+ ". Please try again later."
472472
)
473473

‎shub/plugins/google_build/utils.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -228,7 +228,7 @@ def generate_jwt_token(secret, payload, algorithm="HS256"):
228228
algorithm: the algorithm to use.
229229
"""
230230
# Add an expiration of 8 hours to the payload
231-
expires_in = settings.SREGISTRY_GOOGLE_BUILD_EXPIRE_SECONDS
231+
expires_in = settings.GOOGLE_BUILD_EXPIRE_SECONDS
232232
payload["exp"] = datetime.utcnow() + timedelta(seconds=expires_in)
233233
return jwt.encode(payload, secret, algorithm).decode("utf-8")
234234

‎shub/plugins/google_build/views.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@
3131
DISABLE_BUILD_RECEIVE,
3232
DISABLE_BUILDING,
3333
DISABLE_GITHUB,
34-
SREGISTRY_GOOGLE_BUILD_LIMIT,
34+
GOOGLE_BUILD_LIMIT,
3535
)
3636
from shub.settings import VIEW_RATE_LIMIT as rl_rate
3737
from shub.settings import VIEW_RATE_LIMIT_BLOCK as rl_block
@@ -211,7 +211,7 @@ def perform_create(self, serializer):
211211
if is_over_limit():
212212
message = (
213213
"Registry concurrent build limit is "
214-
+ "%s" % SREGISTRY_GOOGLE_BUILD_LIMIT
214+
+ "%s" % GOOGLE_BUILD_LIMIT
215215
+ ". Please try again later."
216216
)
217217
print(message)

0 commit comments

Comments
 (0)