Hmmm. I wonder why this happened. The commands that are invoked for the test are meant to be mocked so that this doesn't happen, see here: https://pbrisbin.com/posts/mocking_bash/

I don't think I can just merge this without getting to the bottom of the drift. Otherwise, the next person to run the suite will just hit a diff themselves :/

Oh, I'm sorry I didn't know about the mocking.

Thought it was meant to happen sometime, as those packages evolve and the person who runs the tests has a fresh copy of the package database.

Feel free to close this.
Thanks.

If things were working correctly, the test should've past for you -- so there is a bug to fix somewhere. It's just not expected that we have to maintain this every time anyone runs the tests on a different system, that wouldn't be tenable.

If you're interested in getting the tests to pass for you, I think we'll have to follow the same kind of thing I did here: https://github.com/pbrisbin/downgrade/blob/master/test/bin/act-like#L10

Basically instead of unintentionally bypassing the mock, if there's some circumstance where a real command is going to be invoked it should fail outright.

If you're interested in getting the tests to pass for you, I think we'll have to follow the same kind of thing I did here: https://github.com/pbrisbin/downgrade/blob/master/test/bin/act-like#L10

This condition exists here too. Its true that very time I ran make test I got some new fixtures, which sounds normal to make the tests fail.

The only change to get the error for missing fixtures and make the test fail outright, was running tests like this:

FIXTURES_NO_RECORD=1  make test

At first, it would be nice if the CI job definition and make test could share the same env variables to avoid frustrations between the two environments.

About the missing features, if I'm correct the features are created based on the parameters passed to the command to be faked, right? Maybe those options now somehow differ and its not under the control of the mocking?

At first, it would be nice if the CI job definition and make test could share the same env variables to avoid frustrations between the two environments.

Anything to avoid frustrations sounds good. Are you suggesting FIXTURES_NO_RECORD=1 by default when you run make test? I'm in favor of that.

[fixtures] are created based on the parameters passed to the command to be faked, right? Maybe those options now somehow differ and its not under the control of the mocking?

That sounds most likely. A test failure caused by FIXTURES_NO_RECORD=1 could report the arguments it saw, so that you debug why something has changed.

Are you suggesting FIXTURES_NO_RECORD=1 by default when you run make test? I'm in favor of that.

Yes, I support that.

A test failure caused by FIXTURES_NO_RECORD=1 could report the arguments it saw, so that you debug why something has changed.

An invocation of curl makes tests fail, but I can't say why. If stdin was also stored in the fixture, it could help. This is the output:

+  No fixture found for
--silent --fail https://aur.archlinux.org/rpc/?v=5&type=info
&arg\[\]=python-guzzle-sphinx-theme
&arg\[\]=rxvt-unicode-patched
&arg\[\]=aurget
&arg\[\]=ruby-install
&arg\[\]=python2-keyczar
&arg\[\]=python-botocore
&arg\[\]=python-behave
&arg\[\]=neovim-symlinks
&arg\[\]=s3cmd-git
&arg\[\]=aws-cli
&arg\[\]=google-cloud-sdk
&arg\[\]=codeclimate
&arg\[\]=libtinfo
&arg\[\]=tmate
&arg\[\]=ruby-kramdown-man
&arg\[\]=python-s3transfer
&arg\[\]=ruby-papertrail
&arg\[\]=laptop-mode-tools
&arg\[\]=xrectsel
&arg\[\]=csslint
&arg\[\]=cram
&arg\[\]=vbump-git
&arg\[\]=python-parse_type
&arg\[\]=google-talkplugin
&arg\[\]=python-bcdoc
&arg\[\]=drync
&arg\[\]=neovim-git
&arg\[\]=downgrade
&arg\[\]=heroku-client-standalone
&arg\[\]=python2-notify2
&arg\[\]=rcm
&arg\[\]=ghu
&arg\[\]=lnav
&arg\[\]=chruby
&arg\[\]=byzanz
and recording is disabled

I added the line feeds to make the snippet look prettier.

I looked into this a bit and couldn't figure it out. I went ahead and committed your updated assertion and fixture, along with some changes that should help us avoid this in the future. See #69 for more details.

If you rebase your other branch(es) over latest master, you should (:crossed_fingers:) find passing tests and no real commands running.