Orderby terms not working properly

I want to print my color_swatches on the frontend. This works fine; however, they get ordered incorrectly (and not as they are displayed in the backend). I’m using the orderby => menu_order, but it dosen’t work either. How do I get the terms ordered by my custom order (as they are ordered in backend).

function ea_display_color_swatches_on_category() {
    $html_swatch = '<div class="swatches-wrapper">';

    if (! wp_is_mobile() ) {
        $terms = wp_get_object_terms( get_the_ID(), 'pa_farve', array( 'orderby' => 'menu_order' ) );

        foreach ( $terms as $term ) {
            $hex_color = get_woocommerce_term_meta( $term->term_id, 'pa_farve_swatches_id_color', true );

            $html_swatch .= '<a class="swatch-color-category tooltips"
                title="' . $term->name . '"
                style="background:' . $hex_color . '" ></a>';
        }

        // end color-swatches-wrapper
        $html_swatch .= '</div>';

        echo $html_swatch;
    }
}
add_action( 'woocommerce_before_shop_loop_item', 'ea_display_color_swatches_on_category' );