Skip to content

Allow IPC namespace to be shared between containers or with the host#8211

Closed
rhatdan wants to merge 1 commit intomoby:masterfrom
rhatdan:shm
Closed

Allow IPC namespace to be shared between containers or with the host#8211
rhatdan wants to merge 1 commit intomoby:masterfrom
rhatdan:shm

Conversation

@rhatdan
Copy link
Copy Markdown
Contributor

@rhatdan rhatdan commented Sep 24, 2014

Some workloads rely on IPC for communications with other processes. We
would like to split workloads between two container but still allow them
to communicate though shared IPC.

This patch mimics the --net code to allow --ipc=host to not split off
the IPC Namespace. ipc=container:CONTAINERID to share ipc between containers

Docker-DCO-1.1-Signed-off-by: Dan Walsh dwalsh@redhat.com (github: rhatdan)

@rhatdan
Copy link
Copy Markdown
Contributor Author

rhatdan commented Sep 24, 2014

I realize that this pull request needs to be split to contribute to libcontainer separately. But I wanted to get an overall feel for the patch before I submit changes to libcontainer.

@crosbymichael
Copy link
Copy Markdown
Contributor

+1 from me

@jeremyeder
Copy link
Copy Markdown

Thanks @rhatdan, I will test and report back.

@SvenDowideit
Copy link
Copy Markdown
Contributor

I'd love to see a little more info, BUT

Docs LGTM - @jamtur01 @fredlf @ostezer

@jamtur01
Copy link
Copy Markdown
Contributor

@rhatdan Could we something in the run docs just saying why you might do this? Or perhaps even a simple example?

@thockin
Copy link
Copy Markdown
Contributor

thockin commented Sep 25, 2014

+1 from me.
On Sep 24, 2014 10:23 AM, "rhatdan" notifications@github.com wrote:

I realize that this pull request needs to be split to contribute to
libcontainer separately. But I wanted to get an overall feel for the patch
before I submit changes to libcontainer.

Reply to this email directly or view it on GitHub
#8211 (comment).

@rhatdan
Copy link
Copy Markdown
Contributor Author

rhatdan commented Sep 25, 2014

@jeremyeder Could you give an example, since you were the trigger for this code.

@rhatdan
Copy link
Copy Markdown
Contributor Author

rhatdan commented Sep 25, 2014

One problem I will have with this patch is that SELinux will not allow it. I might have to change the code to say ipc=container:CONTAINERID implies that the SELinux labels of the new container has to match the label of CONTAINERID. Also ipc=host would most likely require --security label:disabled. (But that is another patch that has not been merged) :^(

@jeremyeder
Copy link
Copy Markdown

Shared memory segments are used to accelerate inter-process communication at memory speed, rather than, say, through pipes or through the network stack. Shared memory is a commonly used technique used in applications such as databases and their worker threads, along with a large population of custom-built (typically C/OpenMPI, C++/using boost libraries) high performance applications that you will find in the scientific computing and financial services industries.

Without the ability to share memory between containers, this class of high-performance workloads will not be candidates for containerization. To Dan's point about security; processes that share memory are explicitly designed to cooperate, and thus should be grouped together from a security/isolation perspective at least as far as memory.

@rhatdan
Copy link
Copy Markdown
Contributor Author

rhatdan commented Sep 25, 2014

@jamtur01 Is @jeremyeder Message above enough to add?

@jeremyeder
Copy link
Copy Markdown

@jamtur01 when I complete testing of this I will share the example/walkthrough. Please give me a day or 2.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Possessive: container's IPC stack.

@fredlf
Copy link
Copy Markdown
Contributor

fredlf commented Sep 25, 2014

+1 to adding a thoroughly explained example. Thanks.

@jeremyeder
Copy link
Copy Markdown

host# setenforce 0 # for now...
  • Testing --ipc=host mode:
  • Host shows a shared memory segment with 7 pids attached, happens to be from httpd:
host# ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x01128e25 0          root       600        1000       7                       
  • Now run a regular container, and it correctly does NOT see the shared memory segment from the host:
host# docker run -it shm ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
  • Run a container with the new --ipc=host option, and it now sees the shared memory segment from the host httpd:
host# docker run -it --ipc=host shm ipcs -m
------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x01128e25 0          root       600        1000       7                   
  • Testing --ipc=container:CONTAINERID mode:
  • Start a container with a program to create a shared memory segment:
# docker run -it shm bash
root@ed735b2264ac: ~ # shm/shm_server &
[1] 21
root@ed735b2264ac: ~ # ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x0000162e 0          root       666        27         1                       
  • Create a 2nd container correctly shows no shared memory segment from 1st container:
# docker run shm ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
  • Create a 3rd container using the new --ipc=container:CONTAINERID option, now it shows the shared memory segment from the first:
# docker run -it --ipc=container:ed735b2264ac shm ipcs -m
root@9f4b21f2d520: ~ # ipcs -m

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status      
0x0000162e 0          root       666        27         1

@rhatdan
Copy link
Copy Markdown
Contributor Author

rhatdan commented Sep 25, 2014

@fredlf Is this what you were thinking?

@fredlf
Copy link
Copy Markdown
Contributor

fredlf commented Sep 25, 2014

Yes, that's terrific. Examples help users so much. Thank you.

@SvenDowideit
Copy link
Copy Markdown
Contributor

+1 to example :)

@jamtur01
Copy link
Copy Markdown
Contributor

Example LGTM - add away to run.md.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reckon this should be "Invalid IPC ..".

@ewindisch
Copy link
Copy Markdown
Contributor

+1; much desired. The code looks good at a glance.

@crosbymichael
Copy link
Copy Markdown
Contributor

Can you please open a PR on libcontainer for those changes?

@crosbymichael
Copy link
Copy Markdown
Contributor

FYI. If we want this in my 1.3 we should have it merged by first of next week. I would like it in if we finish all the changes.

@thockin
Copy link
Copy Markdown
Contributor

thockin commented Oct 7, 2014

FWIW Go conventions say not to capitalize error messages.
On Oct 7, 2014 6:08 AM, "O.S. Tezer" notifications@github.com wrote:

LGTM.

I think it would be better, however, if we had fmt.Errorf messages
capitalized? (@jamtur01 https://github.com/jamtur01)

Reply to this email directly or view it on GitHub
#8211 (comment).

@jamtur01
Copy link
Copy Markdown
Contributor

jamtur01 commented Oct 7, 2014

https://code.google.com/p/go-wiki/wiki/CodeReviewComments#Error_Strings - not sure how far we've adopted those conventions though.

@ghost
Copy link
Copy Markdown

ghost commented Oct 7, 2014

@thockin @jamtur01 cheers for the heads up, my bad.
I think I've seen some capitalized error messages which got me confused.

@thockin
Copy link
Copy Markdown
Contributor

thockin commented Oct 7, 2014

It is a very minor point in the grand scheme of things :)
On Oct 7, 2014 7:44 AM, "O.S. Tezer" notifications@github.com wrote:

@thockin https://github.com/thockin @jamtur01
https://github.com/jamtur01 cheers for the heads up, my bad.
I think I've seen some capitalized error messages which got me confused.

Reply to this email directly or view it on GitHub
#8211 (comment).

@rhatdan
Copy link
Copy Markdown
Contributor Author

rhatdan commented Oct 7, 2014

@ostezer I just copied these from the error messages for --net, so if we decide to capitilize I would probably change --network stuff also.
@SvenDowideit what do you think?

@SvenDowideit
Copy link
Copy Markdown
Contributor

@rhatdan easy :) not in this PR - and we can leave it for the techwriter review of strings :)

mostly we need core review now.

@ghost
Copy link
Copy Markdown

ghost commented Oct 8, 2014

Thanks @rhatdan ;-)

@rhatdan rhatdan force-pushed the shm branch 2 times, most recently from 2fe6751 to 92ab266 Compare October 9, 2014 13:28
@rhatdan
Copy link
Copy Markdown
Contributor Author

rhatdan commented Oct 9, 2014

Since this is not going to make it into the docker-1.3 pull, I completed the SELinux support.

If you run a container with --ipc=host, this means you could share IPC information with any process on the host OS, so I disable SELinux label enforcement.

If you run a container with --ipc=container:ID we have to run the new container process with the same SELinux labels as the container you are sharing IPC with, or SELinux will block the interaction.

@rhatdan rhatdan force-pushed the shm branch 2 times, most recently from 6b52df8 to 2f4a770 Compare October 9, 2014 14:28
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should be "used by"?

@fredlf
Copy link
Copy Markdown
Contributor

fredlf commented Oct 9, 2014

Docs LGTM once the missing word is added.

@SvenDowideit
Copy link
Copy Markdown
Contributor

Docs are good to go, waiting on Core reviews :) @crosbymichael @tiborvass :)

@rhatdan rhatdan force-pushed the shm branch 6 times, most recently from e12b084 to a77b917 Compare October 23, 2014 22:09
@rhatdan
Copy link
Copy Markdown
Contributor Author

rhatdan commented Oct 23, 2014

Added tests to verify --ipc=host and --ipc=container:ID works properly.

Some workloads rely on IPC for communications with other processes.  We
would like to split workloads between two container but still allow them
to communicate though shared IPC.

This patch mimics the --net code to allow --ipc=host to not split off
the IPC Namespace.  ipc=container:CONTAINERID to share ipc between containers

If you share IPC between containers, then you need to make sure SELinux labels
match.

Docker-DCO-1.1-Signed-off-by: Dan Walsh <dwalsh@redhat.com> (github: rhatdan)
@crosbymichael
Copy link
Copy Markdown
Contributor

Closing in favor of #8835

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants