Do we need to acquire a lock before segment uploads or can we support concurrent uploads as well

As long as segments in the remote store are uploaded and downloaded for backup and restore part, I don't think locking is required. Once we start writing to remote store via IndexWriter, locking would be required.

Open to suggestions.

If we extend BaseDirectory, it will provide locking functionality with the provided LockFactory.

Lets say if we go with concurrent writes to local and remote store would we still need a copyBytes or writeByte

Do we also use ByteBuffer for leveraging I/O optimizations?

Lets say if we go with concurrent writes to local and remote store would we still need a copyBytes or writeByte

For concurrent writes, we will need writeByte/s

Do we also use ByteBuffer for leveraging I/O optimizations?

Right. Current implementation may not be optimized. It also depends on how a given repository implementation is handling writeBlob.
We will be having a continuous performance test setup which will provide insight into the performance issues.

Another case, let's say it's not concurrent, but we want writes to only happen on remote which can be directly searchable we might want to just do writeBytes to remote directly

Right, the idea is to open up the methods as per the requirement. As you said, writeByte/writeBytes methods will be required when IndexWriter is using RemoteDirectory to create the segments (concurrent writes or writing only to remote store).

We also have a VerifyingIndexInput we might want to consider that handles the checksums

Yes, either VerifyingIndexInput or ChecksumIndexInput can be used. I have added it as a ToDo item in the javadoc of the class.

Issue is created for Checksum. Tagging this comment on the issue.