Add support for filtering events and minor related changes by aalemayhu · Pull Request #673 · cilium/cilium

aalemayhu · 2017-05-09T14:12:06Z

The new filtering support consists of --type, --to and --from.
The cilium help monitor output now looks like this:

The monitor displays notifications and events omitted by the BPF
programs attached to endpoints and devices. This includes:
  * Dropped packet notifications
  * Captured packet traces
  * Debugging information

Usage:
  cilium monitor [flags]

Flags:
  -d, --dissect             Dissect packet data
      --from uint16         filter by source endpoint id
  -c, --num-cpus int        Number of CPUs (default 2)
  -n, --num-pages int       Number of pages for ring buffer (default 64)
      --related-to uint32   filter by either source or destination endpoint id
      --to uint32           filter by destination endpoint id
  -t, --type string         Filter by event types [drop debug capture]

Global Flags:
      --config string   config file (default is $HOME/.cilium.yaml)
  -D, --debug           Enable debug messages
  -H, --host string     URI to server-side API

Some example usage:

cilium monitor --type drop will only show you messages for packets being
dropped.
cilium monitor --type drop --from 173 is same as above but will only match
when the source has the provided id.
cilium monitor --type drop --to 123 should match when we get a drop event
and the destination has the provided id.
cilium monitor --type drop --related-to 749 shows drop notifications where
the source or destination has the provided id.

Closes: #473 (Feature: Add filtering to cilium monitor)

tgraf

Overall this is great. It would be even better if we can support a --related-to ID which takes an endpoint ID and then prints everything which has the endpoint ID in either the from to to. This would show everything in and out of a particular endpoint.

tgraf · 2017-05-09T16:16:12Z

I would rename this to validateEventTypeFilter

tgraf · 2017-05-09T16:20:09Z

Consider writing this as a map instead:

eventTypes := map[string]int { "debug": bpfdebug.MessageTypeDebug, "drop": bpfdebug.MessageTypeDrop, "capture": bpfdebug.MessageTypeCapture, }

This also makes it clear which names corresponds to which message type without depending on order

tgraf · 2017-05-09T16:21:30Z

You can then just check the type against the map:

if eventTypeIdx, ok = eventTypes[eventType]; !ok { err := "Unknown type (%s). Please use one of the following ones %s\n" [...] }

tgraf · 2017-05-09T16:23:30Z

The > is misleading, write this as:

if eventTypeIdx != bpfdebug.MessageTypeUnspec && messageType != eventTypeIdx {

@scanf was this addressed?

@aanm meant my comment above which is still not addressed. It's a cosmetic thing though.

@aanm ah, must have left this on a different machine. It was marked as outdated so I just assumed it was addressed since Github automatically collapses these. Will follow up. Thanks!

tgraf · 2017-05-09T16:24:10Z

Initialize them to uint16 so you don't have to cast later on:

fromSource = uint16(0)

tgraf · 2017-05-09T21:02:40Z

This is fine, if you want to generate it:

types := []string{} for _, v := range eventTypes { types = append(types, v) } return types

aanm

Great additions. I've one question. Is it possible do multiple filters at the same time?

aanm · 2017-05-09T21:54:27Z

filter - > Filter

The same goes with the next 2 liness

aanm · 2017-05-09T21:54:48Z

Why Uint16Var and the others have Uint32Var?

To match the existing message format. Using other types would generate compile
time errors. The source fields from the message types define the
.Source format as uint16.

See
https://github.com/cilium/cilium/blob/master/pkg/bpfdebug/drop.go#L30
https://github.com/cilium/cilium/blob/master/pkg/bpfdebug/debug.go#L129
https://github.com/cilium/cilium/blob/master/pkg/bpfdebug/debug.go#L229

also for DstID or Arg1 (uint32) see

https://github.com/cilium/cilium/blob/master/pkg/bpfdebug/drop.go#L36
https://github.com/cilium/cilium/blob/master/pkg/bpfdebug/debug.go#L131
https://github.com/cilium/cilium/blob/master/pkg/bpfdebug/debug.go#L233

The ctypes are in

https://github.com/cilium/cilium/blob/master/bpf/lib/common.h
https://github.com/cilium/cilium/blob/master/bpf/lib/dbg.h

but @borkmann and @tgraf can probably give a much better explanation on why
those fields are uint16 and uint32.

aanm · 2017-05-09T21:57:36Z

Shouldn't this be related > 0 && uint16(related) != src && related != dst?

aalemayhu · 2017-05-09T23:25:18Z

@aanm yes combining options should work fine, but please try out. If you discover something doesn't work. Let me know, so I can fix and add a test for it. Thanks.

aanm · 2017-05-10T07:40:08Z

@scanf How does the to flag works? I've been using it but I don't see any packets showing up, only if I use the from flag.

aalemayhu · 2017-05-10T08:08:03Z

@aanm --to option uses the .Arg1 or .DstID which is what I thought
destination is supposed to be. Can you show an example of how you are using it?
Did you enable the relevant configuration you are interested in cilium config Debug=true DropNotification=true Policy=true?

aanm · 2017-05-10T08:21:40Z

@scanf why Policy needs to be set to true?

tgraf · 2017-05-10T13:15:54Z

Passing dm.Arg1 as destination ID is wrong. In most cases, that is not the destination. We can't support --to for debug messages. Let's just pass in0 for now.

tgraf · 2017-05-10T13:17:33Z

Same here, we would need to extend the BPF progs to always specify the destination ID before we can pass that in. dc.Arg1 will mean something different depending on the actual message being sent.

This will be used for testing the new filtering arguments in a follow up commit. Related-to: #473 (Feature: Add filtering to cilium monitor) Signed-off-by: Alexander Alemayhu <alexander@alemayhu.com>

monitor_resume is useful because it behaves the same as monitor_start() but instead does not overwrite DUMP_FILE. This will be used in follow up to test multiple runs of the monitor. Signed-off-by: Alexander Alemayhu <alexander@alemayhu.com>

ianvernon · 2017-05-10T18:15:07Z

The ordering seems incorrect here - it should be:
fmt.Printf("CPU: %02d: Lost %d events\n", cpu, lost.Lost)

ianvernon · 2017-05-10T18:16:27Z

Documentation needed for this function.

ianvernon · 2017-05-10T18:16:56Z

Documentation needed for this function.

ianvernon · 2017-05-10T18:18:11Z

Documentation needed for this function.

ianvernon · 2017-05-10T18:18:47Z

Documentation needed for this function.

ianvernon

Mainly nits on my part, but please add the requested documentation.

We are already printing it in func receiveEvent(...). To be consistent print it here as well, and maybe this will be useful for future debugging. Signed-off-by: Alexander Alemayhu <alexander@alemayhu.com>

The new filtering support consists of `--type`, `--to` and `--from`. The `cilium help monitor` output now looks like this: The monitor displays notifications and events omitted by the BPF programs attached to endpoints and devices. This includes: * Dropped packet notifications * Captured packet traces * Debugging information Usage: cilium monitor [flags] Flags: -d, --dissect Dissect packet data --from uint16 filter by source endpoint id -c, --num-cpus int Number of CPUs (default 2) -n, --num-pages int Number of pages for ring buffer (default 64) --related-to uint32 filter by either source or destination endpoint id --to uint32 filter by destination endpoint id -t, --type string Filter by event types [drop debug capture] Global Flags: --config string config file (default is $HOME/.cilium.yaml) -D, --debug Enable debug messages -H, --host string URI to server-side API Some example usage: - `cilium monitor --type drop` will only show you messages for packets being dropped. - `cilium monitor --type drop --from 173` is same as above but will only match when the source has the provided id. - `cilium monitor --type drop --to 123` should match when we get a drop event and the destination has the provided id. - `cilium monitor --type drop --related-to 749` shows drop notifications where the source or destination has the provided id. Closes: #473 (Feature: Add filtering to cilium monitor) Signed-off-by: Alexander Alemayhu <alexander@alemayhu.com>

aalemayhu · 2017-05-10T19:20:17Z

@aanm sorry, the policy part can be ignored. Please try again to see if you still have issues.

aalemayhu · 2017-05-10T19:20:38Z

@ianvernon fixed, let me know what to reword. Thanks!

aanm · 2017-05-12T20:54:27Z

@scanf was this addressed?

aalemayhu · 2017-05-13T06:05:50Z

@aanm I am not sure what you mean in the last comment. Could you elaborate on what should have been addressed? Thanks.

Let's merge as-is, we can fix stlye afterwards.

aanm · 2017-05-13T11:41:28Z

@scanf I was referring this comment #673 (comment)

Thomas says: >The > is misleading Related-to: #673 (Add support for filtering events and minor related changes) Suggested-by: Thomas Graf <thomas@cilium.io> Suggested-by: André Martins <andre@cilium.io> Signed-off-by: Alexander Alemayhu <alexander@alemayhu.com>

As the previous commit mentions, this does not occur on main, v1.16, and v1.15 due to cilium#29036. However, in v1.14 and v1.13, we need to take special care because the NameManager and SelectorCache lock can be taken while the Endpoint lock is held during Endpoint deletion. Here are the relevant stacktraces regarding the deadlock: ``` 1: sync.Mutex.Lock [75 minutes] [Created by http.(*Server).Serve in goroutine 1699 @ server.go:3086] sync sema.go:77 runtime_SemacquireMutex(*uint32(0x5), false, 43690) sync mutex.go:171 (*Mutex).lockSlow(*Mutex(cilium#1733)) sync mutex.go:90 (*Mutex).Lock(...) sync rwmutex.go:147 (*RWMutex).Lock(*RWMutex(0xb0)) fqdn name_manager.go:70 (*NameManager).Lock(0xffffffffffffffff) policy selectorcache.go:964 (*SelectorCache).RemoveSelectors(cilium#1088, {cilium#28569, 0xb, 1}, {cilium#643, cilium#32582}) policy l4.go:810 (*L4Filter).removeSelectors(cilium#32582, cilium#29992) policy l4.go:817 (*L4Filter).detach(cilium#719, cilium#29993) policy l4.go:988 L4PolicyMap.Detach(...) policy l4.go:1179 (*L4Policy).Detach(cilium#20318, cilium#1383) policy resolve.go:103 (*selectorPolicy).Detach(...) policy distillery.go:81 (*PolicyCache).delete(cilium#1354, cilium#19354) policy distillery.go:138 (*PolicyCache).LocalEndpointIdentityRemoved(cilium#523, cilium#1163) identitymanager manager.go:167 (*IdentityManager).remove(cilium#706, cilium#19354) identitymanager manager.go:147 (*IdentityManager).Remove(cilium#706, cilium#19354) identitymanager manager.go:52 Remove(...) endpoint endpoint.go:1146 (*Endpoint).leaveLocked(cilium#1883, cilium#12221, {0x30, 0}) endpoint endpoint.go:2192 (*Endpoint).Delete(cilium#1883, {0x80, 0xaa}) endpointmanager manager.go:380 (*EndpointManager).removeEndpoint(cilium#1161, 0, {0xff, 0xff}) endpointmanager manager.go:394 (*EndpointManager).RemoveEndpoint(...) cmd endpoint.go:684 (*Daemon).deleteEndpointQuiet(...) cmd endpoint.go:666 (*Daemon).deleteEndpoint(cilium#1155, cilium#1883) cmd endpoint.go:713 (*Daemon).DeleteEndpoint(cilium#1155, {cilium#27399, cilium#8108}) cmd endpoint.go:770 (*deleteEndpointID).Handle(cilium#21193, {cilium#2452, {cilium#27399, 0x4d}}) endpoint delete_endpoint_id.go:66 (*DeleteEndpointID).ServeHTTP(cilium#5934, {cilium#666, cilium#2242}, cilium#2452) middleware operation.go:28 (*Context).RoutesHandler.NewOperationExecutor.func1({cilium#666, cilium#2242}, cilium#2452) http server.go:2136 HandlerFunc.ServeHTTP(ReadCloser(cilium#130), func{cilium#2242, 0x3}) middleware router.go:78 NewRouter.func1({cilium#666, cilium#2242}, cilium#2451) http server.go:2136 HandlerFunc.ServeHTTP(ReadCloser(cilium#718), func{cilium#2242, #59}) middleware redoc.go:72 Redoc.func1({cilium#666, cilium#2242}, cilium#1251) http server.go:2136 HandlerFunc.ServeHTTP(ReadCloser(cilium#4920), func{cilium#2242, #45}) middleware spec.go:46 Spec.func1({cilium#666, cilium#2242}, cilium#4921) http server.go:2136 HandlerFunc.ServeHTTP(ReadCloser(cilium#10532), func{cilium#2242, cilium#23015}) metrics middleware.go:64 (*APIEventTSHelper).ServeHTTP(cilium#1459, {cilium#668, cilium#10533}, cilium#2451) api apipanic.go:42 (*APIPanicHandler).ServeHTTP(cilium#722, {cilium#668, cilium#10533}, cilium#4922) http server.go:2938 serverHandler.ServeHTTP(*Server(cilium#8105), cilium#668, cilium#10533, 0x6) http server.go:2009 (*conn).serve(*conn(cilium#16005), Context{cilium#673, cilium#1554}) 8: sync.Mutex.Lock [74 minutes] [Created by http.(*Server).Serve in goroutine 1699 @ server.go:3086] sync sema.go:77 runtime_SemacquireMutex(*, 0x47, cilium#1154) sync mutex.go:171 (*Mutex).lockSlow(cilium#706) sync mutex.go:90 (*Mutex).Lock(...) sync rwmutex.go:147 (*RWMutex).Lock(*) identitymanager manager.go:99 (*IdentityManager).RemoveOldAddNew(cilium#706, 0, cilium#1154) identitymanager manager.go:123 RemoveOldAddNew(...) endpoint policy.go:852 (*Endpoint).SetIdentity(*, cilium#1154, 0) endpoint endpoint.go:1932 (*Endpoint).identityLabelsChanged(*, {cilium#674, *}, 1) endpoint endpoint.go:1780 (*Endpoint).runIdentityResolver(*, {cilium#674, *}, 1, 1) endpoint endpoint.go:1720 (*Endpoint).UpdateLabels(*, {cilium#674, *}, *, *, 8) cmd endpoint.go:477 (*Daemon).createEndpoint(cilium#1155, {cilium#673, *}, {cilium#683, cilium#1155}, *) cmd endpoint.go:542 (*putEndpointID).Handle(cilium#21192, {*, *, {*, 0xe}}) endpoint put_endpoint_id.go:58 (*PutEndpointID).ServeHTTP(cilium#3961, {cilium#666, *}, *) middleware operation.go:28 (*Context).RoutesHandler.NewOperationExecutor.func1({cilium#666, *}, *) http server.go:2136 HandlerFunc.ServeHTTP(*, {cilium#666, *}, *) middleware router.go:78 NewRouter.func1({cilium#666, *}, *) http server.go:2136 HandlerFunc.ServeHTTP(*, {cilium#666, *}, *) middleware redoc.go:72 Redoc.func1({cilium#666, *}, cilium#1251) http server.go:2136 HandlerFunc.ServeHTTP(*, {cilium#666, *}, #45) middleware spec.go:46 Spec.func1({cilium#666, *}, *) http server.go:2136 HandlerFunc.ServeHTTP(*, {cilium#666, *}, *) metrics middleware.go:64 (*APIEventTSHelper).ServeHTTP(cilium#1459, {cilium#668, *}, *) api apipanic.go:42 (*APIPanicHandler).ServeHTTP(#49, {cilium#668, *}, *) http server.go:2938 serverHandler.ServeHTTP({cilium#653}, {cilium#668, *}, 6) http server.go:2009 (*conn).serve(*, {cilium#673, cilium#1554}) 5: sync.Mutex.Lock [75 minutes] [Created by eventqueue.(*EventQueue).Run in goroutine 1482 @ eventqueue.go:229] sync sema.go:77 runtime_SemacquireMutex(cilium#142, 0xe8, *) sync mutex.go:171 (*Mutex).lockSlow(cilium#1733) sync mutex.go:90 (*Mutex).Lock(...) sync rwmutex.go:147 (*RWMutex).Lock(0x68) fqdn name_manager.go:70 (*NameManager).Lock(*) policy selectorcache.go:798 (*SelectorCache).AddFQDNSelector(cilium#1088, {cilium#643, *}, {{*, 0x4d}, {0, 0}}) policy l4.go:628 (*L4Filter).cacheFQDNSelector(...) policy l4.go:623 (*L4Filter).cacheFQDNSelectors(*, {*, 4, cilium#193}, cilium#536) policy l4.go:725 createL4Filter({cilium#680, *}, {*, 1, 1}, 0, {cilium#660, *}, {{*, 4}, ...}, ...) policy l4.go:879 createL4EgressFilter(...) policy rule.go:717 mergeEgressPortProto({cilium#680, *}, #44, {*, 0xa, 0}, *, {cilium#660, *}, {{*, ...}, ...}, ...) policy rule.go:672 mergeEgress.func1({cilium#660, *}) api l4.go:284 PortRules.Iterate({*, 1, cilium#546}, *) policy rule.go:624 mergeEgress({cilium#680, *}, *, {*, 1, 1}, 0, {cilium#661, *}, {cilium#662, ...}, ...) policy rule.go:753 (*rule).resolveEgressPolicy(*, {cilium#680, *}, *, *, *, {0, 0, 0}, {0, ...}) policy rules.go:103 ruleSlice.resolveL4EgressPolicy({*, *, *}, {cilium#680, *}, *) policy repository.go:718 (*Repository).resolvePolicyLocked(cilium#1089, *) policy distillery.go:119 (*PolicyCache).updateSelectorPolicy(cilium#1354, *) policy distillery.go:153 (*PolicyCache).UpdatePolicy(...) endpoint policy.go:262 (*Endpoint).regeneratePolicy(*) endpoint bpf.go:744 (*Endpoint).runPreCompilationSteps(*, *, *) endpoint bpf.go:589 (*Endpoint).regenerateBPF(*, *) endpoint policy.go:457 (*Endpoint).regenerate(*, *) endpoint events.go:53 (*EndpointRegenerationEvent).Handle(*, *) eventqueue eventqueue.go:245 (*EventQueue).run.func1() sync once.go:74 (*Once).doSlow(*, *) sync once.go:65 (*Once).Do(...) eventqueue eventqueue.go:233 (*EventQueue).run(*) 1: select [75 minutes] [Created by eventqueue.(*EventQueue).Run in goroutine 1482 @ eventqueue.go:229] semaphore semaphore.go:60 (*Weighted).Acquire(cilium#1092, {cilium#671, cilium#722}, cilium#766) lock semaphored_mutex.go:30 (*SemaphoredMutex).Lock(...) ipcache ipcache.go:140 (*IPCache).Lock(...) ipcache cidr.go:56 (*IPCache).AllocateCIDRs(cilium#1316, {0, 0, cilium#2037}, {0, 0, 0}, 0) ipcache cidr.go:103 (*IPCache).AllocateCIDRsForIPs(0, {0, cilium#697, 0}, 0xffffffffffffffff) cmd identity.go:114 cachingIdentityAllocator.AllocateCIDRsForIPs(...) policy selectorcache.go:509 (*SelectorCache).allocateIdentityMappings(cilium#1088, {{0, 0}, {cilium#5036, 0x2b}}, #45) policy selectorcache.go:843 (*SelectorCache).AddFQDNSelector(cilium#1088, {cilium#643, cilium#27811}, {{0, 0}, {cilium#5036, 0x2b}}) policy l4.go:628 (*L4Filter).cacheFQDNSelector(...) policy l4.go:623 (*L4Filter).cacheFQDNSelectors(cilium#27811, {cilium#42936, 0x287, cilium#193}, cilium#536) policy l4.go:725 createL4Filter({cilium#680, cilium#22826}, {cilium#22828, 1, 1}, 0, {cilium#660, cilium#24510}, {{cilium#4487, 3}, ...}, ...) policy l4.go:879 createL4EgressFilter(...) policy rule.go:717 mergeEgressPortProto({cilium#680, cilium#22826}, #44, {cilium#22828, 0xa, #78536}, #79633, {cilium#660, cilium#24510}, {{cilium#4487, ...}, ...}, ...) policy rule.go:672 mergeEgress.func1({cilium#660, cilium#24510}) api l4.go:284 PortRules.Iterate({cilium#24510, 1, cilium#546}, cilium#11741) policy rule.go:624 mergeEgress({cilium#680, cilium#22826}, cilium#18687, {cilium#22828, 1, 1}, 0, {cilium#661, cilium#5624}, {cilium#662, ...}, ...) policy rule.go:753 (*rule).resolveEgressPolicy(cilium#24575, {cilium#680, cilium#22826}, cilium#18687, cilium#29345, cilium#4782, {0, 0, 0}, {0, ...}) policy rules.go:103 ruleSlice.resolveL4EgressPolicy({cilium#10690, 0xb, 5}, {cilium#680, cilium#22826}, cilium#18687) policy repository.go:718 (*Repository).resolvePolicyLocked(cilium#1089, cilium#18461) policy distillery.go:119 (*PolicyCache).updateSelectorPolicy(cilium#1354, cilium#18461) policy distillery.go:153 (*PolicyCache).UpdatePolicy(...) endpoint policy.go:262 (*Endpoint).regeneratePolicy(cilium#1748) endpoint bpf.go:744 (*Endpoint).runPreCompilationSteps(cilium#1748, cilium#27542, cilium#4781) endpoint bpf.go:589 (*Endpoint).regenerateBPF(cilium#1748, cilium#27542) endpoint policy.go:457 (*Endpoint).regenerate(cilium#1748, cilium#27542) endpoint events.go:53 (*EndpointRegenerationEvent).Handle(cilium#18609, cilium#703) eventqueue eventqueue.go:245 (*EventQueue).run.func1() sync once.go:74 (*Once).doSlow(*Once(cilium#45414), func(cilium#69)) sync once.go:65 (*Once).Do(...) eventqueue eventqueue.go:233 (*EventQueue).run(cilium#6023) ``` Generated from pp tool: https://github.com/maruel/panicparse Signed-off-by: Chris Tarazi <chris@isovalent.com>

aalemayhu requested a review from tgraf May 9, 2017 14:12

tgraf suggested changes May 9, 2017

View reviewed changes

tgraf reviewed May 9, 2017

View reviewed changes

aanm requested changes May 9, 2017

View reviewed changes

tgraf suggested changes May 10, 2017

View reviewed changes

aalemayhu added 2 commits May 10, 2017 16:19

tests: use arguments if passed in to monitor_start

f529680

This will be used for testing the new filtering arguments in a follow up commit. Related-to: #473 (Feature: Add filtering to cilium monitor) Signed-off-by: Alexander Alemayhu <alexander@alemayhu.com>

tests: new function monitor_resume

c55aa99

monitor_resume is useful because it behaves the same as monitor_start() but instead does not overwrite DUMP_FILE. This will be used in follow up to test multiple runs of the monitor. Signed-off-by: Alexander Alemayhu <alexander@alemayhu.com>

ianvernon reviewed May 10, 2017

View reviewed changes

ianvernon suggested changes May 10, 2017

View reviewed changes

aalemayhu added 2 commits May 10, 2017 21:01

cmd/monitor: print CPU when losing events

5001e1c

We are already printing it in func receiveEvent(...). To be consistent print it here as well, and maybe this will be useful for future debugging. Signed-off-by: Alexander Alemayhu <alexander@alemayhu.com>

aalemayhu self-assigned this May 11, 2017

tgraf approved these changes May 12, 2017

View reviewed changes

tgraf requested review from aanm and ianvernon May 12, 2017 20:25

aanm previously requested changes May 12, 2017

View reviewed changes

Comment thread cilium/cmd/monitor.go Outdated

Copy link
Copy Markdown

Member

aanm May 12, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@scanf was this addressed?

ianvernon approved these changes May 12, 2017

View reviewed changes

tgraf merged commit bc0d0e5 into cilium:master May 13, 2017

aalemayhu deleted the filter-473 branch May 13, 2017 11:44

aalemayhu mentioned this pull request May 13, 2017

cmd/monitor: be more specific with eventType match #700

Merged

aalemayhu removed their assignment Jan 16, 2018

Conversation

aalemayhu commented May 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tgraf left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aanm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aalemayhu commented May 9, 2017

Uh oh!

aanm commented May 10, 2017

Uh oh!

aalemayhu commented May 10, 2017

Uh oh!

aanm commented May 10, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ianvernon left a comment

Choose a reason for hiding this comment

Uh oh!

aalemayhu commented May 10, 2017

Uh oh!

aalemayhu commented May 10, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aalemayhu commented May 13, 2017

Uh oh!

aanm commented May 13, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

aalemayhu commented May 9, 2017 •

edited

Loading