[JIT][tracer] Sanity checks for tracing by jamesr66a · Pull Request #10841 · pytorch/pytorch

jamesr66a · 2018-08-24T00:21:02Z

TODO: integrate into torch.onnx.export -- separate PR

Problem: We have a facility to trace PyTorch operations on Python code, but there are several failure modes where the trace is not representative of the actual underlying computation:

The tracer encountered dynamic control flow
Some computation escaped the tracer, and appeared as a Constant tensor node in the graph
Some stateful function was traced, e.g. someone did an optimization in Python by memoizing function outputs

Objective: In an ideal world, this whole process would be automated and the user can trust that the system will magically capture the intended semantics from the program. Realistically speaking, we will likely have to settle with a human-in-the-loop error reporting system, allowing for the user to identify problems and modify the source code to allow for tracing.

Stage 1 (this PR): Output-level checking & graph diff. torch.jit.trace gains a kwarg 'check_inputs', which is a list of tuples of input arguments. We will iterate through the list and trace the function again for each set of check inputs. We'll also interpret the original trace with these inputs and compare output values and graphs, printing a diff of the graph if there is a difference.

Examples:

@torch.jit.trace(torch.rand(3, 4), check_inputs=[(torch.rand(4, 5),)])
def foo(x):
    y = torch.arange(0, x.shape[0]).float()
    return x + y.unsqueeze(1)

torch.jit.TracingCheckError: Tracing failed sanity checks!
ERROR: Graphs differed across invocations!
	Graph diff:
		  graph(%0 : Dynamic) {
		-   %1 : Dynamic = prim::Constant[value= 0  1  2 [ CPULongType{3} ]]()
		?                                                              ^
		+   %1 : Dynamic = prim::Constant[value= 0  1  2  3 [ CPULongType{4} ]]()
		?                                                +++              ^
		    %2 : int = prim::Constant[value=0]()
		    %3 : Dynamic = aten::_cast_Float(%1, %2)
		    %4 : int = prim::Constant[value=1]()
		    %5 : Dynamic = aten::unsqueeze(%3, %4)
		    %6 : int = prim::Constant[value=1]()
		    %7 : Dynamic = aten::add(%0, %5, %6)
		    return (%7);
		  }
	Node diff:
		- %1 : Dynamic = prim::Constant[value= 0  1  2 [ CPULongType{3} ]]()
		?                                                            ^
		+ %1 : Dynamic = prim::Constant[value= 0  1  2  3 [ CPULongType{4} ]]()
		?                                              +++              ^
	Trace source location:
		dank.py(5): foo
		/Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(402): wrapper
		dank.py(3): <module>
	Check source location:
		dank.py(5): foo
		/Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(281): check_trace
		/Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(408): wrapper
		dank.py(3): <module>
ERROR: Tensor-valued Constant nodes differed in value across invocations. This often indicates that the tracer has encountered untraceable code.
	Node:
		%1 : Dynamic = prim::Constant[value= 0  1  2 [ CPULongType{3} ]]()
	Source Location:
		dank.py(5): foo
		/Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(402): wrapper
		dank.py(3): <module>
	Comparison exception:
		Not equal to tolerance rtol=1e-07, atol=0

		(shapes (3,), (4,) mismatch)
		 x: array([0, 1, 2])
		 y: array([0, 1, 2, 3])

==

@torch.jit.trace(torch.rand(3, 4), check_inputs=[(torch.rand(3, 4),)])
def foo(x):
    y = x.data
    return x + y

torch.jit.TracingCheckError: Tracing failed sanity checks!
ERROR: Traced function outputs do not match the Python function outputs.
ERROR: Tensor-valued Constant nodes differed in value across invocations. This often indicates that the tracer has encountered untraceable code.
	Node:
		%1 : Dynamic = prim::Constant[value=<Tensor>]()
	Source Location:
		dank.py(6): foo
		/Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(402): wrapper
		dank.py(3): <module>
	Comparison exception:
		Not equal to tolerance rtol=1e-07, atol=0

		(mismatch 100.0%)
		 x: array([0.397137, 0.956105, 0.169478, 0.560292, 0.392568, 0.108441,
		       0.97645 , 0.34412 , 0.951246, 0.793061, 0.557595, 0.770245],
		      dtype=float32)
		 y: array([0.243178, 0.315964, 0.972041, 0.0215  , 0.927751, 0.457512,
		       0.951092, 0.97883 , 0.048688, 0.118066, 0.779345, 0.271272],
		      dtype=float32)

==

import torch

@torch.jit.trace(torch.rand(3, 4), check_inputs=[(torch.rand(4, 4),)])
def foo(x):
    for _ in range(x.size(0)):
        x = torch.neg(x)
    return x

torch.jit.TracingCheckError: Tracing failed sanity checks!
ERROR: Traced function outputs do not match the Python function outputs.
ERROR: Graphs differed across invocations!
	Graph diff:
		  graph(%0 : Dynamic) {
		    %1 : Dynamic = aten::neg(%0)
		    %2 : Dynamic = aten::neg(%1)
		    %3 : Dynamic = aten::neg(%2)
		+   %4 : Dynamic = aten::neg(%3)
		-   return (%3);
		?            ^
		+   return (%4);
		?            ^
		  }

==

import torch

def foo(x):
    if not hasattr(foo, 'cache'):
        foo.cache = torch.neg(x)
    return x + foo.cache

traced = torch.jit.trace(torch.rand(3, 4), check_inputs=[(torch.rand(3, 4),)])(foo)

torch.jit.TracingCheckError: Tracing failed sanity checks!
ERROR: Traced function outputs do not match the Python function outputs.
ERROR: Graphs differed across invocations!
	Graph diff:
		  graph(%0 : Dynamic) {
		-   %1 : Dynamic = aten::neg(%0)
		+   %1 : Dynamic = prim::Constant[value=<Tensor>]()
		    %2 : int = prim::Constant[value=1]()
		    %3 : Dynamic = aten::add(%0, %1, %2)
		    return (%3);
		  }
	Node diff:
		- %1 : Dynamic = aten::neg(%0)
		+ %1 : Dynamic = prim::Constant[value=<Tensor>]()
	Trace source location:
		test.py(5): foo
		/Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(402): wrapper
		test.py(8): <module>
	Check source location:
		test.py(6): foo
		/Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(281): check_trace
		/Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(408): wrapper
		test.py(8): <module>

The following two examples show instances where program semantics are lost in the Python -> trace transformation, and repeated invocation does not give us useful debug information. Further design in underway for catching these scenarios.

import torch

@torch.jit.trace(torch.rand(3, 4), check_inputs=[(torch.rand(3, 4),)])
def foo(x):
    for i in range(3):
        x[i, :] = torch.zeros(4)
    return x

torch.jit.TracingCheckError: Tracing failed sanity checks!
ERROR: Traced function outputs do not match the Python function outputs.
Exception:
Not equal to tolerance rtol=1e-07, atol=0

(mismatch 100.0%)
 x: array([0.830221, 0.915481, 0.940281, 0.555241], dtype=float32)
 y: array([0., 0., 0., 0.], dtype=float32)

==

import torch

@torch.jit.trace(torch.rand(3, 4), check_inputs=[(torch.rand(5, 6),)])
def foo(x):
    x.view(-1).add_(-x.view(-1))
    return x

torch.jit.TracingCheckError: Tracing failed sanity checks!
ERROR: Traced function outputs do not match the Python function outputs.
Exception:
Not equal to tolerance rtol=1e-07, atol=0

(mismatch 100.0%)
 x: array([0.734441, 0.445327, 0.640592, 0.30076 , 0.891674, 0.124771],
      dtype=float32)
 y: array([0., 0., 0., 0., 0., 0.], dtype=float32)

facebook-github-bot

jamesr66a has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

jamesr66a has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

jamesr66a has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

jamesr66a has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot

jamesr66a has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

test/expect/TestScript.test_call_traced_module_from_traced_module.expect

  %3 : Double(3, 5) = aten::mm(%0, %1), scope: TracedModule
  %4 : Double(3, 7) = aten::mm(%3, %2), scope: TracedModule/TracedModule[TracedModule1][mod]
-  %5 : Long() = prim::Constant[value={1}](), scope: TracedModule
+  %5 : Double() = prim::Constant[value={1}](), scope: TracedModule


test/test_jit.py

        @torch.jit.trace(torch.rand(3, 4))
        def traced_fn(x):
-            return pm(x) + 1
+            return pm(x) + 1.0


test/test_jit.py

+        def foo(x):
+            return torch.dropout(x, p=0.5, train=False)
+
+        np.testing.assert_allclose(foo(input), input)


torch/csrc/jit/ir.cpp

+  if (kind() == aten::dropout && is_constant(attr::train) && !get<bool>(attr::train).value()) {
+    return false;
+  }
+  // batch_norm with training = False is deterministic


torch/csrc/jit/ir.cpp

+}  // namespace
+
+bool Node::isNondeterministic() const {
+  if (nondeterminstic_aten_ops().count(kind()) == 0) {


torch/jit/__init__.py

+                    try:
+                        mod_tensor_val = n_mod.t('value')
+                        check_tensor_val = n_check.t('value')
+                    except RuntimeError:


torch/jit/__init__.py

+            tensor_compare_errors = None
+            # Check Tensor-valued constant nodes
+            for n_mod, n_check in zip(mod_canonicalized.nodes(), check_canonicalized.nodes()):
+                if n_mod.kind() == n_check.kind() and n_mod.kind() == 'prim::Constant':


torch/jit/__init__.py

+        try:
+            traced_outs = wrap_non_iterable(module(*_clone_inputs(inputs)))
+            # TODO: multi-level nested compare of results.
+            traced_outs = [out for out in traced_outs if isinstance(out, torch.Tensor)]


torch/jit/__init__.py

+                iter(x)
+            except TypeError:
+                x = [x]
+            return x


torch/jit/__init__.py

+
+        check_inputs = kwargs.pop('check_inputs', None)
+        disable_checks = kwargs.pop('disable_checks', False)
+        check_tolerance = kwargs.pop('check_tolerance', 1e-7)


zdevito

I don't have additional things beside what Adam said except for nit.

We should consider setting the tolerance low enough that we don't cause anyone false positive. It would be a pretty bad experience to realize the only reason tracing failed was due to numeric nondeterminism.

torch/csrc/jit/passes/decay_types.cpp

+void DecayTypes(const std::shared_ptr<Graph>& graph) {
+  for (Value * input : graph->inputs()) {
+    if (input->type()->cast<TensorType>()) {
+      input->setType(DynamicType::get());


torch/csrc/jit/python_ir.cpp

      ss << n;
      return ss.str();
    })
+    .def("sourceLocation", [](Node & n) {


torch/csrc/jit/python_ir.cpp

+      std::stringstream ss;
+      if (auto sl = n.getSourceLocation())
+        sl->highlight(ss);
+      return ss.str();


torch/jit/__init__.py

+
+
+# Check the traced module against a set of user-provided validation inputs
+def check_trace(check_inputs, func, executor_options, module, check_tolerance):


torch/jit/__init__.py

+def check_trace(check_inputs, func, executor_options, module, check_tolerance):
+    for inputs in check_inputs:
+        check_mod = TopLevelTracedModule(func, **executor_options)
+        check_mod._create_method_from_trace('forward', func, _clone_inputs(inputs))


torch/jit/__init__.py

+                    try:
+                        mod_tensor_val = n_mod.t('value')
+                        check_tensor_val = n_check.t('value')
+                    except RuntimeError:


torch/jit/__init__.py

+
+        check_inputs = kwargs.pop('check_inputs', None)
+        disable_checks = kwargs.pop('disable_checks', False)
+        check_tolerance = kwargs.pop('check_tolerance', 1e-7)


facebook-github-bot

jamesr66a has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

torch/csrc/jit/register_prim_ops.cpp

+        prim::NoneGenerator,
+        [](Node* node) {
+          return [](Stack& stack) {
+            stack.push_back(at::Tensor());


torch/jit/__init__.py

+        except Exception as e:
+            msg = 'Encountered an exception while running checking trace with check inputs.\nException:\n' \
+                + indent(str(e))
+            raise TracingCheckError(*graph_diagnostic_info(), extra_msg=msg)


torch/csrc/jit/ir.cpp

+    "aten::normal(Tensor mean, float std, *, Tensor generator) -> Tensor",
+    "aten::poisson(Tensor self, Tensor generator) -> Tensor",
+    "aten::rrelu(Tensor self, Scalar lower, Scalar upper, int training, Tensor generator) -> Tensor",
+    "aten::rrelu_with_noise(Tensor self, Tensor noise, Scalar lower, Scalar upper, int training, Tensor generator) -> Tensor"


zdevito

Probably should add aten::rand to the non-deterministic things. I am approving anyway so that it won't block landing.

jamesr66a · 2018-08-29T00:21:27Z

@pytorchbot retest this please

facebook-github-bot

jamesr66a has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Summary: TODO: integrate into torch.onnx.export -- separate PR *Problem:* We have a facility to trace PyTorch operations on Python code, but there are several failure modes where the trace is not representative of the actual underlying computation: * The tracer encountered dynamic control flow * Some computation escaped the tracer, and appeared as a Constant tensor node in the graph * Some stateful function was traced, e.g. someone did an optimization in Python by memoizing function outputs *Objective*: In an ideal world, this whole process would be automated and the user can trust that the system will magically capture the intended semantics from the program. Realistically speaking, we will likely have to settle with a human-in-the-loop error reporting system, allowing for the user to identify problems and modify the source code to allow for tracing. *Stage 1* (this PR): Output-level checking & graph diff. torch.jit.trace gains a kwarg 'check_inputs', which is a list of tuples of input arguments. We will iterate through the list and trace the function again for each set of check inputs. We'll also interpret the original trace with these inputs and compare output values and graphs, printing a diff of the graph if there is a difference. Examples: ``` torch.jit.trace(torch.rand(3, 4), check_inputs=[(torch.rand(4, 5),)]) def foo(x): y = torch.arange(0, x.shape[0]).float() return x + y.unsqueeze(1) ``` ``` torch.jit.TracingCheckError: Tracing failed sanity checks! ERROR: Graphs differed across invocations! Graph diff: graph(%0 : Dynamic) { - %1 : Dynamic = prim::Constant[value= 0 1 2 [ CPULongType{3} ]]() ? ^ + %1 : Dynamic = prim::Constant[value= 0 1 2 3 [ CPULongType{4} ]]() ? +++ ^ %2 : int = prim::Constant[value=0]() %3 : Dynamic = aten::_cast_Float(%1, %2) %4 : int = prim::Constant[value=1]() %5 : Dynamic = aten::unsqueeze(%3, %4) %6 : int = prim::Constant[value=1]() %7 : Dynamic = aten::add(%0, %5, %6) return (%7); } Node diff: - %1 : Dynamic = prim::Constant[value= 0 1 2 [ CPULongType{3} ]]() ? ^ + %1 : Dynamic = prim::Constant[value= 0 1 2 3 [ CPULongType{4} ]]() ? +++ ^ Trace source location: dank.py(5): foo /Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(402): wrapper dank.py(3): <module> Check source location: dank.py(5): foo /Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(281): check_trace /Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(408): wrapper dank.py(3): <module> ERROR: Tensor-valued Constant nodes differed in value across invocations. This often indicates that the tracer has encountered untraceable code. Node: %1 : Dynamic = prim::Constant[value= 0 1 2 [ CPULongType{3} ]]() Source Location: dank.py(5): foo /Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(402): wrapper dank.py(3): <module> Comparison exception: Not equal to tolerance rtol=1e-07, atol=0 (shapes (3,), (4,) mismatch) x: array([0, 1, 2]) y: array([0, 1, 2, 3]) ``` == ``` torch.jit.trace(torch.rand(3, 4), check_inputs=[(torch.rand(3, 4),)]) def foo(x): y = x.data return x + y ``` ``` torch.jit.TracingCheckError: Tracing failed sanity checks! ERROR: Traced function outputs do not match the Python function outputs. ERROR: Tensor-valued Constant nodes differed in value across invocations. This often indicates that the tracer has encountered untraceable code. Node: %1 : Dynamic = prim::Constant[value=<Tensor>]() Source Location: dank.py(6): foo /Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(402): wrapper dank.py(3): <module> Comparison exception: Not equal to tolerance rtol=1e-07, atol=0 (mismatch 100.0%) x: array([0.397137, 0.956105, 0.169478, 0.560292, 0.392568, 0.108441, 0.97645 , 0.34412 , 0.951246, 0.793061, 0.557595, 0.770245], dtype=float32) y: array([0.243178, 0.315964, 0.972041, 0.0215 , 0.927751, 0.457512, 0.951092, 0.97883 , 0.048688, 0.118066, 0.779345, 0.271272], dtype=float32) ``` == ``` import torch torch.jit.trace(torch.rand(3, 4), check_inputs=[(torch.rand(4, 4),)]) def foo(x): for _ in range(x.size(0)): x = torch.neg(x) return x ``` ``` torch.jit.TracingCheckError: Tracing failed sanity checks! ERROR: Traced function outputs do not match the Python function outputs. ERROR: Graphs differed across invocations! Graph diff: graph(%0 : Dynamic) { %1 : Dynamic = aten::neg(%0) %2 : Dynamic = aten::neg(%1) %3 : Dynamic = aten::neg(%2) + %4 : Dynamic = aten::neg(%3) - return (%3); ? ^ + return (%4); ? ^ } ``` == ``` import torch def foo(x): if not hasattr(foo, 'cache'): foo.cache = torch.neg(x) return x + foo.cache traced = torch.jit.trace(torch.rand(3, 4), check_inputs=[(torch.rand(3, 4),)])(foo) ``` ``` torch.jit.TracingCheckError: Tracing failed sanity checks! ERROR: Traced function outputs do not match the Python function outputs. ERROR: Graphs differed across invocations! Graph diff: graph(%0 : Dynamic) { - %1 : Dynamic = aten::neg(%0) + %1 : Dynamic = prim::Constant[value=<Tensor>]() %2 : int = prim::Constant[value=1]() %3 : Dynamic = aten::add(%0, %1, %2) return (%3); } Node diff: - %1 : Dynamic = aten::neg(%0) + %1 : Dynamic = prim::Constant[value=<Tensor>]() Trace source location: test.py(5): foo /Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(402): wrapper test.py(8): <module> Check source location: test.py(6): foo /Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(281): check_trace /Users/jamesreed/onnx-fairseq/pytorch/torch/jit/__init__.py(408): wrapper test.py(8): <module> ``` The following two examples show instances where program semantics are lost in the Python -> trace transformation, and repeated invocation does not give us useful debug information. Further design in underway for catching these scenarios. ``` import torch torch.jit.trace(torch.rand(3, 4), check_inputs=[(torch.rand(3, 4),)]) def foo(x): for i in range(3): x[i, :] = torch.zeros(4) return x ``` ``` torch.jit.TracingCheckError: Tracing failed sanity checks! ERROR: Traced function outputs do not match the Python function outputs. Exception: Not equal to tolerance rtol=1e-07, atol=0 (mismatch 100.0%) x: array([0.830221, 0.915481, 0.940281, 0.555241], dtype=float32) y: array([0., 0., 0., 0.], dtype=float32) ``` == ``` import torch torch.jit.trace(torch.rand(3, 4), check_inputs=[(torch.rand(5, 6),)]) def foo(x): x.view(-1).add_(-x.view(-1)) return x ``` ``` torch.jit.TracingCheckError: Tracing failed sanity checks! ERROR: Traced function outputs do not match the Python function outputs. Exception: Not equal to tolerance rtol=1e-07, atol=0 (mismatch 100.0%) x: array([0.734441, 0.445327, 0.640592, 0.30076 , 0.891674, 0.124771], dtype=float32) y: array([0., 0., 0., 0., 0., 0.], dtype=float32) ``` Pull Request resolved: pytorch#10841 Differential Revision: D9499945 Pulled By: jamesr66a fbshipit-source-id: 1f842a32d0b0645259cc43b29700b86d99c59a45

jamesr66a requested review from apaszke, colesbury, ezyang, gchanan, soumith and zdevito as code owners August 24, 2018 00:21

jamesr66a force-pushed the trace_check branch 4 times, most recently from 0f4a2cb to 538be05 Compare August 24, 2018 17:00

facebook-github-bot reviewed Aug 24, 2018

View reviewed changes

James Reed added 5 commits August 27, 2018 10:09

Simple trace checking functionality

4fdcf71

Updates and fixes

67e46ad

Tolerances for GPU fusion tests

a25e5fe

Print out exception when we fail to run the network

6a8e46b

Dumb fix

3516276

jamesr66a force-pushed the trace_check branch from f12e9f1 to 3516276 Compare August 27, 2018 17:14

facebook-github-bot reviewed Aug 27, 2018

View reviewed changes

jamesr66a changed the title ~~[WIP] Sanity checks for tracing~~ [JIT][tracer] Sanity checks for tracing Aug 27, 2018

Handle nondeterministic nodes

dd7bd71

jamesr66a force-pushed the trace_check branch from d6b368b to dd7bd71 Compare August 27, 2018 19:24

facebook-github-bot reviewed Aug 27, 2018

View reviewed changes

apaszke reviewed Aug 27, 2018

View reviewed changes

zdevito reviewed Aug 27, 2018

View reviewed changes

jamesr66a force-pushed the trace_check branch 3 times, most recently from addf644 to 4300eaf Compare August 27, 2018 23:52

Address comments

9fb01a7

jamesr66a force-pushed the trace_check branch from 4300eaf to 9fb01a7 Compare August 27, 2018 23:53

Support Generator type in schema

289c249

zou3519 added oncall: jit Add this issue/PR to JIT oncall triage queue 0.3 labels Aug 28, 2018

jamesr66a removed the 0.3 label Aug 28, 2018

forward executor_options

3d749fe

facebook-github-bot reviewed Aug 28, 2018

View reviewed changes

apaszke reviewed Aug 28, 2018

View reviewed changes

zdevito reviewed Aug 28, 2018

View reviewed changes

zdevito approved these changes Aug 28, 2018

View reviewed changes

Add rand ops

dc008cc

facebook-github-bot reviewed Aug 29, 2018

View reviewed changes

facebook-github-bot closed this in beeec47 Aug 29, 2018

ezyang added the merged label Jun 26, 2019

lara-hdr mentioned this pull request Aug 28, 2019

torch.arange always generate constant result in tracing #20075

Closed



		# Check the traced module against a set of user-provided validation inputs
		def check_trace(check_inputs, func, executor_options, module, check_tolerance):

Conversation

jamesr66a commented Aug 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

zdevito left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

facebook-github-bot left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

This comment was marked as off-topic.

Uh oh!

zdevito left a comment

Choose a reason for hiding this comment

Uh oh!

jamesr66a commented Aug 24, 2018 •

edited

Loading