When running a matrix animation I see just translation animation taking place, no scaling or rotating. This happens on Android (S7 Edge and Nexus 5) and iOS (Simulator iP7+) Stackoverflow link. What it looks like for me:

Code for reproducing it:

import 'package:flutter/material.dart';
import 'package:vector_math/vector_math_64.dart' as vector;

void main() {
  runApp(new MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return new MaterialApp(
      title: 'Flutter Demo',
      theme: new ThemeData(
        primarySwatch: Colors.blue,
      ),
      home: new MyHomePage(),
    );
  }
}

class MyHomePage extends StatefulWidget {
  MyHomePage({Key key}) : super(key: key);

  @override
  _MyHomePageState createState() => new _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> with TickerProviderStateMixin{
  static Matrix4 originalTransformation = new Matrix4.compose(
      new vector.Vector3(1.0, 1.0, 1.0),
      new vector.Quaternion.euler(0.0, 0.0, 0.0),
      new vector.Vector3(1.0, 1.0, 1.0));

  static Matrix4 animatedTransformation = new Matrix4.compose(
      new vector.Vector3(5.0, 260.0, 1.0),
      new vector.Quaternion.euler(0.0, 1.0, -0.7),
      new vector.Vector3(0.6, 0.6, 0.6));

  Matrix4 currentMatrix = originalTransformation;
  AnimationController animationController;
  Matrix4Tween animationTween;

  @override
  Widget build(BuildContext context) {
    return new Scaffold(
      resizeToAvoidBottomPadding: true,
      body: new Column(
        mainAxisSize: MainAxisSize.max,
        mainAxisAlignment: MainAxisAlignment.center,
        children: [
          new AnimatedBuilder(
            // Pass animation controller
            animation: animationController,
            builder: (BuildContext context, Widget child) => new Container(
              // Apply transformation
              transform: animationTween.evaluate(animationController),
              child: child,
            ),
            // Passing child argument
            child: new GestureDetector(
              onTap: _onClick,
              child: new Container(
                height: 400.0,
                width: 400.0,
                color: Colors.red,
              ),
            ),
          ),
        ],
      ),
    );
  }

  _onClick() {
    // Start animation
    animationController.forward(from: 0.0);
    return;
  }

  @override
  void initState() {
    super.initState();

    // Init tween for matrix
    animationTween = new Matrix4Tween(
        begin: originalTransformation,
        end: animatedTransformation
    );

    // Init animation controller
    animationController = new AnimationController(
      vsync: this,
      duration: new Duration(milliseconds: 800),
    );
  }
}

Logs

Flutter Run

Launching lib/main.dart on SM G935F in debug mode...
Running 'gradle assembleDebug'... 2.3s
Built build/app/outputs/apk/app-debug.apk (21.7MB).
Syncing files to device SM G935F...
D/ViewRootImpl@56af48cMainActivity: ViewPostImeInputStage processPointer 0
W/System (25448): ClassLoader referenced unknown path: /system/framework/QPerformance.jar
E/BoostFramework(25448): BoostFramework() : Exception_1 = java.lang.ClassNotFoundException: Didn't find class "com.qualcomm.qti.Performance" on path: DexPathList[[],nativeLibraryDirectories=[/system/lib, /vendor/lib]]
V/BoostFramework(25448): BoostFramework() : mPerf = null
D/ViewRootImpl@56af48cMainActivity: ViewPostImeInputStage processPointer 1

🔥 To hot reload your app on the fly, press "r". To restart the app entirely, press "R".
An Observatory debugger and profiler on SM G935F is available at: http://127.0.0.1:8104/
For a more detailed help message, press "h". To quit, press "q".
D/ViewRootImpl@56af48cMainActivity: ViewPostImeInputStage processPointer 0
D/ViewRootImpl@56af48cMainActivity: ViewPostImeInputStage processPointer 1

Flutter Analyze

Analyzing /Users/snoopy/Documents/github/ColorsProject...
No issues found!
Ran in 5.8s

Flutter Doctor

[✓] Flutter (on Mac OS X 10.12.5 16F73, locale en-HR, channel alpha)
• Flutter at /Users/snoopy/flutter
• Framework revision 6f77b4a (2 weeks ago), 2017-06-14 17:27:08 -0700
• Engine revision 18fdfb8
• Tools Dart version 1.24.0-dev.6.7

[✓] Android toolchain - develop for Android devices (Android SDK 25.0.3)
• Android SDK at /Users/snoopy/Library/Android/sdk
• Platform android-25, build-tools 25.0.3
• ANDROID_HOME = /Users/snoopy/Library/Android/sdk
• Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 1.8.0_112-release-b06)

[-] iOS toolchain - develop for iOS devices (Xcode 8.3.3)
• Xcode at /Applications/Xcode.app/Contents/Developer
• Xcode 8.3.3, Build version 8E3004b
✗ libimobiledevice and ideviceinstaller are not installed or require updating. To update, run:
brew update
brew uninstall --ignore-dependencies libimobiledevice
brew install --HEAD libimobiledevice
brew install ideviceinstaller
• ios-deploy 1.9.1
• CocoaPods version 1.2.1

[✓] Android Studio (version 2.3)
• Android Studio at /Applications/Android Studio.app/Contents
• Gradle version 3.2
• Java version OpenJDK Runtime Environment (build 1.8.0_112-release-b06)

[✓] IntelliJ IDEA Community Edition (version 2017.1.4)
• Flutter plugin version 14.0
• Dart plugin version 171.4694.29

[✓] Connected devices
• SM G935F • 9885b651454a4a3034 • android-arm • Android 7.0 (API 24) (emulator)