CodeJail Support #284
Description
Is your feature request related to a problem? Please describe.
I was looking into implementing CodeJail (https://github.com/edx/codejail), as it can be an attack vector for tutor instances (and openedx in general), as it´s in charge of running untrusted code from studio and LMS (Ex: Custom Python-Evaluated Input).
Currently CodeJail is not supported in tutor, and is executed on the openedx context, which have a lot more permissions.
I will be out out this month and can´t continue working on this feature, so I will leave what I have done to support it, if anyone can continue (I will be able to continue on march).
Current Work
To implement codejail in the docker environment, first I installed manually the codejail code in docker as a build step, adding the sandbox user to the environment.
RUN useradd -ms /bin/bash sandbox && \
git clone https://github.com/edx/codejail /openedx/sandbox && \
virtualenv /openedx/sandbox/env && \
/openedx/sandbox/env/bin/pip install -r /openedx/sandbox/requirements/sandbox.txt && \
chown -R sandbox:sandbox /openedx/sandbox && \
apt-get update && apt-get install sudo
After this, you must add in the cms.env and lms.env config files the codejail items with the correct paths.
"CODE_JAIL": {
"python_bin": "/openedx/sandbox/env/bin/python2",
"user": "sandbox",
"limits": {
"CPU": 1,
"VMEM": 268435456,
"REALTIME": 3,
"FSIZE": 1048576
}
},
After this, you must add the apparmor profile to jail the code executed by the sandbox (In the host system). I used the original apparmor profile included in the repository, and combined it with the docker-default environment, so we can execute the docker instance with the default security applied. This is done by creating /etc/apparmor.d/containers/docker-edx-sandbox with the following content, and activating it by running sudo apparmor_parser -r -W /etc/apparmor.d/containers/docker-edx-sandbox
#include <tunables/global>
profile docker-edx-sandbox flags=(attach_disconnected,mediate_deleted) {
#include <abstractions/base>
network,
capability,
file,
umount,
signal (receive) peer=unconfined,
signal (receive) peer=snap.docker.dockerd,
signal (send,receive) peer=docker-edx-sandbox,
deny @{PROC}/* w, # deny write for all files directly in /proc (not in a subdir)
# deny write to files not in /proc/<number>/** or /proc/sys/**
deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9]*}/** w,
deny @{PROC}/sys/[^k]** w, # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w, # deny everything except shm* in /proc/sys/kernel/
deny @{PROC}/sysrq-trigger rwklx,
deny @{PROC}/kcore rwklx,
deny mount,
deny /sys/[^f]*/** wklx,
deny /sys/f[^s]*/** wklx,
deny /sys/fs/[^c]*/** wklx,
deny /sys/fs/c[^g]*/** wklx,
deny /sys/fs/cg[^r]*/** wklx,
deny /sys/firmware/** rwklx,
deny /sys/kernel/security/** rwklx,
ptrace (trace,read,tracedby,readby) peer=docker-edx-sandbox,
/openedx/sandbox/env/bin/python2 Cx -> child,
profile child flags=(attach_disconnected,mediate_deleted){
#include <abstractions/base>
#include <abstractions/python>
/openedx/sandbox/env/** mr,
/tmp/codejail-*/ rix,
/tmp/codejail-*/** wrix,
allow /dev/pts/0 rw,
}
}
Finally, you must setup the security_opt of the docker instances by adding in the docker-compose the following line (in lms, cms, lms_worker, cms_worker)
cms:
...
security_opt:
- apparmor:docker-edx-sandbox
After restarting the instances, you will have a sandboxed instance, you can test it by running the following command (Which must output nothing)
# Normal
docker-compose exec cms /bin/sh -c "python -c \"import os; os.system('ls /etc')\""
# Sandbox
docker-compose exec cms /bin/sh -c "/openedx/sandbox/env/bin/python2 -c \"import os; os.system('ls /etc')\""
Current Problems
All tests were done only on the cms context, as I wanted to support one context, but it should apply to the others directly.
- CodeJail did´t detect the configuration, and I had to run the configuration code (https://github.com/edx/codejail/blob/master/codejail/django_integration.py) manually in the settings file, appending this to the end.
import codejail.jail_code
python_bin = CODE_JAIL.get('python_bin')
if python_bin:
user = CODE_JAIL['user']
codejail.jail_code.configure("python", python_bin, user=user)
limits = CODE_JAIL.get('limits', {})
for name, value in limits.items():
codejail.jail_code.set_limit(name, value)
- When running an xblock example, I get the following error (the lines might be different as I was testing for changes)
cms_1 | 2020-01-31 18:00:37 [10] [DEBUG] GET /xblock/container/block-v1:eol+CC050+2019_02+type@vertical+block@36d78f9070f64369aa9d8e1dfad9058e
cms_1 | 2020-01-31 18:00:42 [10] [DEBUG] POST /event
cms_1 | 2020-01-31 18:00:42 [10] [DEBUG] POST /preview/xblock/block-v1:eol+CC050+2019_02+type@problem+block@88f87468890c43cfbbaf03104e7be87c/handler/xmodule_handler/problem_check
cms_1 | 2020-01-31 18:00:42,772 INFO 10 [codejail] subproc.py:46 - Executed jailed code 88f87468890c43cfbbaf03104e7be87c in /tmp/codejail-3KyIQv, with PID 225
cms_1 | 2020-01-31 18:00:42,795 ERROR 10 [capa.capa_problem] capa_problem.py:886 - Error while execing script code:
cms_1 |
cms_1 | def test_add_to_ten(expect, ans):
cms_1 | return test_add(10, ans)
cms_1 |
cms_1 |
cms_1 |
cms_1 | def test_add(expect, ans):
cms_1 | try:
cms_1 | a1=int(ans[0])
cms_1 | a2=int(ans[1])
cms_1 | return (a1+a2) == int(expect)
cms_1 | except ValueError:
cms_1 | return False
cms_1 |
cms_1 | Traceback (most recent call last):
cms_1 | File "/openedx/edx-platform/common/lib/capa/capa/capa_problem.py", line 883, in _extract_context
cms_1 | unsafely=self.capa_system.can_execute_unsafe_code(),
cms_1 | File "/openedx/edx-platform/common/lib/capa/capa/safe_exec/safe_exec.py", line 157, in safe_exec
cms_1 | raise e
cms_1 | SafeExecException: Couldn't execute jailed code: stdout: '', stderr: 'sudo: unable to execute /openedx/sandbox/env/bin/python2: Resource temporarily unavailable\n' with status code: -1
cms_1 | 2020-01-31 18:00:42,795 WARNING 10 [edx.courseware] capa_base.py:266 - cannot create LoncapaProblem block-v1:eol+CC050+2019_02+type@problem+block@88f87468890c43cfbbaf03104e7be87c: Error while executing script code: Couldn't execute jailed code: stdout: '', stderr: 'sudo: unable to execute /openedx/sandbox/env/bin/python2: Resource temporarily unavailable\n' with status code: -1
Solving these two problems should be enough to have something ready to integrate into tutor, and will be a lot more secure as a production setup. I will answer any question I can (If they don´t shutdown my computer at work with everything working)
Best,
Felipe.