[fdborch] Store vlan id of FDB entry instead of port pvid into stateD… by jipanyang · Pull Request #759 · sonic-net/sonic-swss

jipanyang · 2019-01-16T03:17:00Z

…B FDB table.

Signed-off-by: Jipan Yang jipan.yang@alibaba-inc.com

What I did
Address issue:
#758

Why I did it
For port in tagged vlan, fdb data stored in stateDB has wrong vlan id.

How I verified it
VS test.

Details if related

jipanyang · 2019-01-16T03:29:44Z

Tried to expand test_fdb_warm.py with case of port in both tagged and untagged vlan, but not able to get tagged packet through the vs veth pair tunnel, somewhere the tag got stripped off before reaching the other side. Not sure if people are aware of that or have solution to the problem.

Had tag 8 (8100 0008) when in sw-srv18 namespace,

$ sudo ip netns exec sw-srv18 tcpdump -i eth0 -xx -vvv
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes



03:03:00.348713 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 8.8.8.1 tell 8.8.8.6, length 28
	0x0000:  ffff ffff ffff 8ecd 6ead ff52 8100 0008
	0x0010:  0806 0001 0800 0604 0001 8ecd 6ead ff52
	0x0020:  0808 0806 0000 0000 0000 0808 0801

tag 8 is missing in vs docker:

root@9abb777a5311:/# tcpdump -i Ethernet72 -vvv -x
03:00:39.215926 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has google-public-dns-a.google.com tell 8.8.8.1, length 28
	0x0000:  0001 0800 0604 0001 0242 ac11 0002 0808
	0x0010:  0801 0000 0000 0000 0808 0808

lguohan · 2019-01-16T18:39:38Z

where is the vs test?

jipanyang · 2019-01-16T20:00:22Z

where is the vs test?

Below is the change in vs I was trying to make, but it failed because tagged arp/ping from dvs.server name space arrived at VS docker interface with tag stripped off. Don't know the root cause yet.

diff --git a/tests/conftest.py b/tests/conftest.py
index 823e129..cb57506 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -673,6 +673,12 @@ class DockerVirtualSwitch(object):
         tbl.set("Vlan" + vlan + "|" + interface, fvs)
         time.sleep(1)
 
+    def create_vlan_member_tagged(self, vlan, interface):
+        tbl = swsscommon.Table(self.cdb, "VLAN_MEMBER")
+        fvs = swsscommon.FieldValuePairs([("tagging_mode", "tagged")])
+        tbl.set("Vlan" + vlan + "|" + interface, fvs)
+        time.sleep(1)
+
     def set_interface_status(self, interface, admin_status):
         if interface.startswith("PortChannel"):
             tbl_name = "PORTCHANNEL"
diff --git a/tests/test_fdb_warm.py b/tests/test_fdb_warm.py
index 492fe78..25451b6 100644
--- a/tests/test_fdb_warm.py
+++ b/tests/test_fdb_warm.py
@@ -40,6 +40,14 @@ def test_fdb_notifications(dvs, testlog):
     dvs.create_vlan_member("6", "Ethernet64")
     dvs.create_vlan_member("6", "Ethernet68")
 
+    dvs.create_vlan_member("6", "Ethernet72")
+    dvs.create_vlan_member("6", "Ethernet76")
+
+    dvs.create_vlan("8")
+    dvs.create_vlan_member_tagged("8", "Ethernet72")
+    dvs.create_vlan_member_tagged("8", "Ethernet76")
+
+
     # Get mapping between interface name and its bridge port_id
     time.sleep(2)
     iface_2_bridge_port_id = dvs.get_map_iface_bridge_port_id(dvs.adb)
@@ -54,6 +62,16 @@ def test_fdb_notifications(dvs, testlog):
         [("SAI_BRIDGE_PORT_ATTR_FDB_LEARNING_MODE", "SAI_BRIDGE_PORT_FDB_LEARNING_MODE_HW")])
     assert ok, str(extra)
 
+    ok, extra = dvs.is_table_entry_exists(dvs.adb, 'ASIC_STATE:SAI_OBJECT_TYPE_BRIDGE_PORT',
+        iface_2_bridge_port_id["Ethernet72"],
+        [("SAI_BRIDGE_PORT_ATTR_FDB_LEARNING_MODE", "SAI_BRIDGE_PORT_FDB_LEARNING_MODE_HW")])
+    assert ok, str(extra)
+
+    ok, extra = dvs.is_table_entry_exists(dvs.adb, 'ASIC_STATE:SAI_OBJECT_TYPE_BRIDGE_PORT',
+        iface_2_bridge_port_id["Ethernet76"],
+        [("SAI_BRIDGE_PORT_ATTR_FDB_LEARNING_MODE", "SAI_BRIDGE_PORT_FDB_LEARNING_MODE_HW")])
+    assert ok, str(extra)
+
     # check fdb aging attr
     ok, extra = dvs.all_table_entry_has_no(dvs.adb, 'ASIC_STATE:SAI_OBJECT_TYPE_SWITCH',
         ".*",
@@ -62,13 +80,24 @@ def test_fdb_notifications(dvs, testlog):
     # bring up vlan and member
     dvs.set_interface_status("Vlan6", "up")
     dvs.add_ip_address("Vlan6", "6.6.6.1/24")
+    dvs.add_ip_address("Vlan8", "8.8.8.1/24")
     dvs.set_interface_status("Ethernet64", "up")
     dvs.set_interface_status("Ethernet68", "up")
+    dvs.set_interface_status("Ethernet72", "up")
+    dvs.set_interface_status("Ethernet76", "up")
     dvs.servers[16].runcmd("ifconfig eth0 6.6.6.6/24 up")
     dvs.servers[16].runcmd("ip route add default via 6.6.6.1")
     dvs.servers[17].runcmd("ifconfig eth0 6.6.6.7/24 up")
     dvs.servers[17].runcmd("ip route add default via 6.6.6.1")
 
+    dvs.servers[18].runcmd("vconfig add eth0 8")
+    dvs.servers[18].runcmd("ifconfig eth0.8 8.8.8.6/24 up")
+    dvs.servers[18].runcmd("ip route add default via 8.8.8.1")
+
+    dvs.servers[19].runcmd("vconfig add eth0 8")
+    dvs.servers[19].runcmd("ifconfig eth0.8 8.8.8.7/24 up")
+    dvs.servers[19].runcmd("ip route add default via 8.8.8.1")
+
     # get neighbor and arp entry
     time.sleep(2)
     rc = dvs.servers[16].runcmd("ping -c 1 6.6.6.7")
@@ -76,6 +105,13 @@ def test_fdb_notifications(dvs, testlog):
     rc = dvs.servers[17].runcmd("ping -c 1 6.6.6.6")
     assert rc == 0
 
+    # get neighbor and arp entry
+    time.sleep(2)
+    rc = dvs.servers[18].runcmd("ping -c 1 8.8.8.7")
+    assert rc == 0
+    rc = dvs.servers[19].runcmd("ping -c 1 8.8.8.6")
+    assert rc == 0
+
 
     # check that the FDB entries were inserted into ASIC DB
     ok, extra = dvs.is_fdb_entry_exists(dvs.adb, "ASIC_STATE:SAI_OBJECT_TYPE_FDB_ENTRY",
@@ -93,9 +129,24 @@ def test_fdb_notifications(dvs, testlog):
     )
     assert ok, str(extra)
 
+    ok, extra = dvs.is_fdb_entry_exists(dvs.adb, "ASIC_STATE:SAI_OBJECT_TYPE_FDB_ENTRY",
+                    [],
+                    [("SAI_FDB_ENTRY_ATTR_TYPE", "SAI_FDB_ENTRY_TYPE_DYNAMIC"),
+                     ("SAI_FDB_ENTRY_ATTR_BRIDGE_PORT_ID", iface_2_bridge_port_id["Ethernet72"]),
+                    ]
+    )
+    assert ok, str(extra)
+    ok, extra = dvs.is_fdb_entry_exists(dvs.adb, "ASIC_STATE:SAI_OBJECT_TYPE_FDB_ENTRY",
+                    [],
+                    [("SAI_FDB_ENTRY_ATTR_TYPE", "SAI_FDB_ENTRY_TYPE_DYNAMIC"),
+                     ("SAI_FDB_ENTRY_ATTR_BRIDGE_PORT_ID", iface_2_bridge_port_id["Ethernet76"]),
+                    ]
+    )
+    assert ok, str(extra)
+
     time.sleep(2)
     counter_inserted = dvs.getCrmCounterValue('STATS', 'crm_stats_fdb_entry_used')
-    assert counter_inserted - counter_before == 2
+    assert counter_inserted - counter_before == 4
 
     # check that the FDB entries were inserted into State DB
     ok, extra = dvs.is_table_entry_exists(dvs.sdb, "FDB_TABLE",
@@ -113,6 +164,22 @@ def test_fdb_notifications(dvs, testlog):
     )
     assert ok, str(extra)
 
+    # check that the FDB entries were inserted into State DB, Vlan8 while not Vlan6(untagged) in the key
+    ok, extra = dvs.is_table_entry_exists(dvs.sdb, "FDB_TABLE",
+                    "Vlan8:.*",
+                    [("port", "Ethernet72"),
+                     ("type", "dynamic"),
+                    ]
+    )
+    assert ok, str(extra)
+    ok, extra = dvs.is_table_entry_exists(dvs.sdb, "FDB_TABLE",
+                    "Vlan8:*",
+                    [("port", "Ethernet76"),
+                     ("type", "dynamic"),
+                    ]
+    )
+    assert ok, str(extra)
+
     # enable warm restart
     (exitcode, result) = dvs.runcmd("config warm_restart enable swss")
     assert exitcode == 0
@@ -160,6 +227,22 @@ def test_fdb_notifications(dvs, testlog):
         )
         assert ok, str(extra)
 
+        # check that the FDB entries were inserted into ASIC DB
+        ok, extra = dvs.is_fdb_entry_exists(dvs.adb, "ASIC_STATE:SAI_OBJECT_TYPE_FDB_ENTRY",
+                        [],
+                        [("SAI_FDB_ENTRY_ATTR_TYPE", "SAI_FDB_ENTRY_TYPE_DYNAMIC"),
+                         ("SAI_FDB_ENTRY_ATTR_BRIDGE_PORT_ID", iface_2_bridge_port_id["Ethernet72"]),
+                        ]
+        )
+        assert ok, str(extra)
+        ok, extra = dvs.is_fdb_entry_exists(dvs.adb, "ASIC_STATE:SAI_OBJECT_TYPE_FDB_ENTRY",
+                        [],
+                        [("SAI_FDB_ENTRY_ATTR_TYPE", "SAI_FDB_ENTRY_TYPE_DYNAMIC"),
+                         ("SAI_FDB_ENTRY_ATTR_BRIDGE_PORT_ID", iface_2_bridge_port_id["Ethernet76"]),
+                        ]
+        )
+        assert ok, str(extra)
+
         # check FDB learning mode
         ok, extra = dvs.is_table_entry_exists(dvs.adb, 'ASIC_STATE:SAI_OBJECT_TYPE_BRIDGE_PORT',
             iface_2_bridge_port_id["Ethernet64"],
@@ -170,6 +253,15 @@ def test_fdb_notifications(dvs, testlog):
             [("SAI_BRIDGE_PORT_ATTR_FDB_LEARNING_MODE", "SAI_BRIDGE_PORT_FDB_LEARNING_MODE_HW")])
         assert ok, str(extra)
 
+        ok, extra = dvs.is_table_entry_exists(dvs.adb, 'ASIC_STATE:SAI_OBJECT_TYPE_BRIDGE_PORT',
+            iface_2_bridge_port_id["Ethernet72"],
+            [("SAI_BRIDGE_PORT_ATTR_FDB_LEARNING_MODE", "SAI_BRIDGE_PORT_FDB_LEARNING_MODE_HW")])
+        assert ok, str(extra)
+        ok, extra = dvs.is_table_entry_exists(dvs.adb, 'ASIC_STATE:SAI_OBJECT_TYPE_BRIDGE_PORT',
+            iface_2_bridge_port_id["Ethernet76"],
+            [("SAI_BRIDGE_PORT_ATTR_FDB_LEARNING_MODE", "SAI_BRIDGE_PORT_FDB_LEARNING_MODE_HW")])
+        assert ok, str(extra)
+
         time.sleep(2)
         counter_restarted = dvs.getCrmCounterValue('STATS', 'crm_stats_fdb_entry_used')
         assert counter_inserted == counter_restarted

jipanyang · 2019-01-24T07:41:01Z

Have turned off all the vlan related offload on both server and vs sides, the vlan tag in packet from server to vs still got stripped off. If anyone has idea about this, please share it, thanks.

server

sudo ip netns exec sw-srv18 tcpdump -i eth0 -xxx -vvv
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

....

23:31:00.347337 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has _gateway tell jipan-Inspiron-3668, length 28
	0x0000:  ffff ffff ffff 12a3 3991 143e 8100 0008
	0x0010:  0806 0001 0800 0604 0001 12a3 3991 143e
	0x0020:  0808 0806 0000 0000 0000 0808 0801
23:31:01.371259 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has _gateway tell jipan-Inspiron-3668, length 28
	0x0000:  ffff ffff ffff 12a3 3991 143e 8100 0008
	0x0010:  0806 0001 0800 0604 0001 12a3 3991 143e
	0x0020:  0808 0806 0000 0000 0000 0808 0801

324 packets captured
324 packets received by filter
0 packets dropped by kernel
jipan@jipan-Inspiron-3668:~/work/src/build/vs/sonic-buildimage/src/sonic-swss/tests$ sudo ip netns exec sw-srv18 ethtool -k  eth0 | grep vlan
rx-vlan-offload: off
tx-vlan-offload: off
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-vlan-stag-hw-insert: off
rx-vlan-stag-hw-parse: off
rx-vlan-stag-filter: off [fixed]

VS

root@5721ec7aef56:/# tcpdump -i Ethernet72 -xxx -vv
tcpdump: listening on Ethernet72, link-type EN10MB (Ethernet), capture size 262144 bytes
....

07:31:47.455326 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 8.8.8.1 tell 8.8.8.6, length 28
	0x0000:  ffff ffff ffff 12a3 3991 143e 0806 0001
	0x0010:  0800 0604 0001 12a3 3991 143e 0808 0806
	0x0020:  0000 0000 0000 0808 0801

07:31:48.475435 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 8.8.8.1 tell 8.8.8.6, length 28
	0x0000:  ffff ffff ffff 12a3 3991 143e 0806 0001
	0x0010:  0800 0604 0001 12a3 3991 143e 0808 0806
	0x0020:  0000 0000 0000 0808 0801
^C
1853 packets captured
1853 packets received by filter
0 packets dropped by kernel
root@5721ec7aef56:/#  ethtool -k Ethernet72  | grep vlan
Cannot get device udp-fragmentation-offload settings: Operation not supported
rx-vlan-offload: off [fixed]
tx-vlan-offload: off
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-vlan-stag-hw-insert: off
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
root@5721ec7aef56:/#

lguohan · 2019-01-24T17:47:12Z

@jipanyang , what about the peer link, e.g. eth0 in the server namespace, and ethX in the sonic namespace. Not, the Ethernet port. If eth0 and ethX has, but not Ethernet, then it is virtual switch problem, we can ask @kcudnik to look at.

jipanyang · 2019-01-24T20:03:18Z

@lguohan @kcudnik I mistaken Ethernet72 as the other side of the veth pair, using eth18 in sonic name space and eth0 in server namespace, still seen the tag being stripped off problem:

server

jipan@jipan-Inspiron-3668:~/work/src/build/sonic-buildimage$ sudo ip netns exec sw-srv18   ethtool -k eth0 | grep vlan
rx-vlan-offload: off
tx-vlan-offload: off
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-vlan-stag-hw-insert: off
rx-vlan-stag-hw-parse: off
rx-vlan-stag-filter: off [fixed]

jipan@jipan-Inspiron-3668:~/work/src/build/sonic-buildimage$ sudo ip netns exec sw-srv18  tcpdump -i eth0 -vv -xxx
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes



11:57:57.982588 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has _gateway tell jipan-Inspiron-3668, length 28
	0x0000:  ffff ffff ffff e618 4ae6 7604 8100 0008
	0x0010:  0806 0001 0800 0604 0001 e618 4ae6 7604
	0x0020:  0808 0806 0000 0000 0000 0808 0801
11:57:59.006575 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has _gateway tell jipan-Inspiron-3668, length 28
	0x0000:  ffff ffff ffff e618 4ae6 7604 8100 0008
	0x0010:  0806 0001 0800 0604 0001 e618 4ae6 7604
	0x0020:  0808 0806 0000 0000 0000 0808 0801



jipan@jipan-Inspiron-3668:~/work/src/build/sonic-buildimage$ sudo ip netns exec sw-srv18  ip add show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0.8@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e6:18:4a:e6:76:04 brd ff:ff:ff:ff:ff:ff
    inet 8.8.8.6/24 brd 8.8.8.255 scope global eth0.8
       valid_lft forever preferred_lft forever
    inet6 fe80::e418:4aff:fee6:7604/64 scope link 
       valid_lft forever preferred_lft forever
131: eth0@if130: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e6:18:4a:e6:76:04 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::e418:4aff:fee6:7604/64 scope link 
       valid_lft forever preferred_lft forever

sonic vs

root@a81e75d7bfb5:/# ethtool -k eth18 | grep vlan
Cannot get device udp-fragmentation-offload settings: Operation not supported
rx-vlan-offload: off
tx-vlan-offload: off
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-vlan-stag-hw-insert: off
rx-vlan-stag-hw-parse: off
rx-vlan-stag-filter: off [fixed]
root@a81e75d7bfb5:/# 

root@a81e75d7bfb5:/# tcpdump -i eth18 -xxx -vv
tcpdump: listening on eth18, link-type EN10MB (Ethernet), capture size 262144 bytes
19:36:53.470786 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 8.8.8.1 tell 8.8.8.6, length 28
	0x0000:  ffff ffff ffff e618 4ae6 7604 0806 0001
	0x0010:  0800 0604 0001 e618 4ae6 7604 0808 0806
	0x0020:  0000 0000 0000 0808 0801
19:36:54.494960 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 8.8.8.1 tell 8.8.8.6, length 28
	0x0000:  ffff ffff ffff e618 4ae6 7604 0806 0001
	0x0010:  0800 0604 0001 e618 4ae6 7604 0808 0806
	0x0020:  0000 0000 0000 0808 0801

lguohan · 2019-01-30T19:04:28Z

i setup simple env. I can see the tag on the virtual switch side. eth0
tcpdump -ve -n -i eth1

19:03:33.142309 ce:6b:47:c3:f7:53 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 46: vlan 8, p 0, ethertype ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.1.0.1 tell 10.1.0.0, length 28

lguohan · 2019-02-11T09:21:10Z

@jipanyang , kamil has fix the vlan tag issue in vs, please add the vstest.

…B FDB table. Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>

Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>

jipanyang · 2019-02-12T01:55:47Z

@lguohan vs test for fdb with tagged vlan added.

jipanyang · 2019-02-13T22:23:57Z

@yxieca please help label "Request_for_201811_branch"

#759) * [fdborch] Store vlan id of FDB entry instead of port pvid into stateDB FDB table. Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com> * Expand fdb test with tagged vlan Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>

Merge two vrf configuration to avoid user can't add or delete management VRF through CLI. Co-authored-by: Minkang-Tsai <58835052+Minkang-Tsai@users.noreply.github.com>

…y functions (sonic-net#759) The template version of hgetall() could work with either map or unordered_map.

sonic-net#759) * [fdborch] Store vlan id of FDB entry instead of port pvid into stateDB FDB table. Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com> * Expand fdb test with tagged vlan Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>

jipanyang force-pushed the fdb_vlan_fix branch from 1815021 to 592c251 Compare January 16, 2019 05:01

prsunny approved these changes Jan 16, 2019

View reviewed changes

stcheng added the Bug 🐛 label Jan 16, 2019

stcheng approved these changes Jan 16, 2019

View reviewed changes

jipanyang added 2 commits February 11, 2019 10:18

[fdborch] Store vlan id of FDB entry instead of port pvid into stateD…

8eaec1a

…B FDB table. Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>

Expand fdb test with tagged vlan

647011d

Signed-off-by: Jipan Yang <jipan.yang@alibaba-inc.com>

jipanyang force-pushed the fdb_vlan_fix branch from 592c251 to 647011d Compare February 11, 2019 22:43

yxieca added the Request for 201811 Branch label Feb 19, 2019

yxieca merged commit 8f06b89 into sonic-net:master Feb 20, 2019

jipanyang mentioned this pull request Mar 18, 2019

fdborch: wrong vlan ID may be saved to stateDB FDB_TABLE and restored. #758

Closed

yxieca added the Included in 201811 Branch label Mar 19, 2019

jipanyang deleted the fdb_vlan_fix branch June 10, 2019 23:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fdborch] Store vlan id of FDB entry instead of port pvid into stateD…#759

[fdborch] Store vlan id of FDB entry instead of port pvid into stateD…#759
yxieca merged 2 commits intosonic-net:masterfrom
jipanyang:fdb_vlan_fix

jipanyang commented Jan 16, 2019

Uh oh!

jipanyang commented Jan 16, 2019

Uh oh!

lguohan commented Jan 16, 2019

Uh oh!

jipanyang commented Jan 16, 2019

Uh oh!

jipanyang commented Jan 24, 2019

Uh oh!

lguohan commented Jan 24, 2019

Uh oh!

jipanyang commented Jan 24, 2019

Uh oh!

lguohan commented Jan 30, 2019

Uh oh!

lguohan commented Feb 11, 2019

Uh oh!

jipanyang commented Feb 12, 2019

Uh oh!

jipanyang commented Feb 13, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

jipanyang commented Jan 16, 2019

Uh oh!

jipanyang commented Jan 16, 2019

Uh oh!

lguohan commented Jan 16, 2019

Uh oh!

jipanyang commented Jan 16, 2019

Uh oh!

jipanyang commented Jan 24, 2019

Uh oh!

lguohan commented Jan 24, 2019

Uh oh!

jipanyang commented Jan 24, 2019

Uh oh!

lguohan commented Jan 30, 2019

Uh oh!

lguohan commented Feb 11, 2019

Uh oh!

jipanyang commented Feb 12, 2019

Uh oh!

jipanyang commented Feb 13, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants