…gl-project#3964)

Signed-off-by: wangyu <wangyu.steph@bytedance.com>

The new session cache has a framework as:

          /<---swap in----\         +------+
         /                 \        |entry0|
    +--------+         +--------+   +------+
    |sessionA|         |sessionX|---+ ...  |   +------------+
    +--------+         +--------+   +------+   |token offset|
        |                  |        |entryX|---+token length|
    +--------+         +--------+   +------+   |connector   |
    |sessionB|         |sessionY|              |uri         |
    +--------+         +--------+              +------------+
        |                  |
    +--------+         +--------+
    |sessionC|         |sessionZ|
    +--------+         +--------+
         \                 /
          \---swap out--->/

1> Use a LRU based list to manage sessions. From GPU HBM to extended
   memory is called 'swap out', the reverse direction is called 'swap
   in'. Then a single sglang instance is able to handle multiple
   sessions.

2> There are one or more 'session cache entry(s)' for a session,
   it's called 'session cache meta'. The session meta could be saved
   into several types of databases:
   a> memkv: a hash map in python. It has the same life cyle as sglang
      program. Once sglang restarts, sglang will lose the previous
      meta data. However, it does *NOT* have any dependence on other
      module.
   b> sglite(in developing). It will persist meta on the machine.
      sglang could load meta after program restarts. Considering that
      the local file based connector in use, it's possible to reload
      all the session meta and data after restarting.
   c> mysql,pgsql,etcd,redis(in developing): distributed database based
      session meta allows multiple sglang instances to share across
      cluster of machines. A request(or a new session) could be
      handled on any sglang instance, the kv-cache is also possible to
      save into/reload from distributed storage system[1], then we can
      benifit:
      * a higher cache hit: because both the session meta and kv-cache
        are saved into distributed database/storage, the session cache
        is expected to hit in a high rate.
      * easy to deploy/manage: sglang instance will be managed by k8s,
        because sglang has already offload the state (meta and kv-cache)
        to distributed system, it becomes *stateless*.
      * the stateless services prove higher CPU utilization from the
        'traditional' web services. We expect this will reproduce on
        the new distributed system.

3> There are token offset, token length, connector and uri fields in
   a single session cache entry.
   The 'connector' may be:
   a> local file directory. It will work fine with sqlite based meta.
   b> valkey, 3fs(and so on): bingo! *stateless* sglang service is
   coming!

Link[1]: commit 1ce4878("feat(remote_model): support variable remote backend for model loader (sgl-project#3964)")
Signed-off-by: Changqi Lu <luchangqi.123@bytedance.com>
Co-authored-by: zhenwei pi <pizhenwei@bytedance.com>
Signed-off-by: zhenwei pi <pizhenwei@bytedance.com>
Signed-off-by: zhenwei pi <zhenwei.pi@linux.dev>