Describe the bug
The Watcher utility can fail to find transactions when transaction volume is particularly high. Here's my best explanation of what's going on:

1. Watcher first tries to find all existing events:

   optimism/packages/core-utils/src/watcher.ts

   Lines 76 to 91 in 1c1e405

   	const blockNumber = await layer.provider.getBlockNumber()
   	const startingBlock = Math.max(blockNumber - this.NUM_BLOCKS_TO_FETCH, 0)
   	const successFilter = {
   	address: layer.messengerAddress,
   	topics: [ethers.utils.id(`RelayedMessage(bytes32)`)],
   	fromBlock: startingBlock,
   	}
   	const failureFilter = {
   	address: layer.messengerAddress,
   	topics: [ethers.utils.id(`FailedRelayedMessage(bytes32)`)],
   	fromBlock: startingBlock,
   	}
   	const successLogs = await layer.provider.getLogs(successFilter)
   	const failureLogs = await layer.provider.getLogs(failureFilter)
   	const logs = successLogs.concat(failureLogs)
   	const matches = logs.filter((log: any) => log.data === msgHash)

2. If the watcher can't find the message relay event in historical blocks, it then installs an event listener until the event is found:

   optimism/packages/core-utils/src/watcher.ts

   Lines 107 to 125 in 1c1e405

   	return new Promise(async (resolve, reject) => {
   	const handleEvent = async (log: any) => {
   	if (log.data === msgHash) {
   	try {
   	const txReceipt = await layer.provider.getTransactionReceipt(
   	log.transactionHash
   	)
   	layer.provider.off(successFilter)
   	layer.provider.off(failureFilter)
   	resolve(txReceipt)
   	} catch (e) {
   	reject(e)
   	}
   	}
   	}
   	layer.provider.on(successFilter, handleEvent)
   	layer.provider.on(failureFilter, handleEvent)
   	})

This causes an issue in the following case:

1. Spam a lot of L1 => L2 transactions.
2. Attempt to wait for L1 => L2 transaction N, assume it hasn't been processed yet because L2 geth is still processing all transactions 0...N-1.
3. Assume the highest block at step (1) above is some M < N. Step (1) will NOT find the event for transaction N.
4. Assume that in between step (1) and step (2), transactions M...P, P > N are processed.
5. Step (2) will NOT find the event for transaction N because the listener is installed AFTER the event has already been emitted.

To Reproduce
Somewhat difficult to reproduce. I discovered this while working on #971. You can try to reproduce by building that branch and executing this test.

Expected behavior
Watcher will find all transactions even in cases of high system load.

Proposed fix
Install the listener BEFORE searching for historical transactions. I think. We just need some new code that 100% guarantees that all blocks will be searched. We could also alternatively NOT make use of the listeners and instead simply scan blocks "the old fashioned way" (in ranges of blocks).