Use case description

Ideally it should be straightforward to use Serverless Framework programmatically. Additionally clean separation of concerns will help significantly the maintenance and will allow to integrate Components engine into the repository.

Improvements towards separation of Packaging and Deployment phases has been moved to #8499

Proposed solution

Step by step let's seclude CLI params and service config resolution logic from a core and layout a new processing flow as follows:

1. If CLI command is -v, output version info and abort
2. If in service context:
   1. Resolve service configuration (with support for -c, --config CLI params), into plain not normalized in any way JSON structure). (If there's parsing error and it's not a CLI help request, crash with meaningful error)
   2. Resolve file, self and strToBool variable sources (but only those not depending on other resolvers)
   3. Ensure that provider and eventual provider.stage properties are fully resolved. (If there's a validation error and it's not a CLI help request, crash with meaningful error on provider being not resolved. Show deprecation and warning notice on provider.stage not being resolved)
   4. Load eventual extra env variables from .env files (If there's parsing error and it's not a CLI help request, crash with meaningful error)
   5. Resolve env and remaining file, self and strToBool variable sources (but only those not depending on other resolvers)
   6. Ensure eventual plugins property is fully resolved.
   7. Initialize plugins (at this stage plugins should have no access to service config or any meta):
      (If there's initialization error and it's not a CLI help request, crash with meaningful error)
      1. Register plugin config schema extensions
      2. Register plugin variable extensions
      3. Register commands and hooks
3. If CLI help request. Display help and abort
4. Parse CLI arguments (with respect to map of supported commands and options)
5. If in service context:
   1. Resolve variables for all variable sources which do not depend on config properties
   2. Resolve all remaining variables in service config
6. Run lifecycle events for CLI command

Additionally to have Framework CLI totally programatic, #1720 has to be addressed.

Implementation spec

Preliminary notes:

• Implementation of each step may break some tests. If those tests relate to functionalities being removed from core they should be simply removed. If it's otherwise ideally if they're fixed with refactoring to runServerless variant (assuming they're applicable for that)
• Ideally if all below points are addressed after following tasks are addressed:
  • Tests refactor: Move all unit tests to "test/unit" folder #8478 - Move all unit tests to test/unit folder
  • CLI: Remove help --verbose option and improve general help output #8497 - Remove CLI help --verbose option and improve general help output

0.1 Unify process error handling

1. Wrap all process logic with try/catch clause
2. Configure handler which generally mirrors one at Error class (but presents simplified logic due to more generic approach). Place handler logic in lib/cli/handle-error.js (and ensure some tests for it)
3. Ensure that uncaughtExceptions are handled with same error handler
4. Remove lib/classes/Error.js logError utility

0.2 Generalize process execution span promise handling (as currently served by serverless.onExitPromise)

1. Reflect execution span of introduced above try/catch clause in promise accessible at lib/cli/execution-span.js (module should export an unresolved promise and method (to be used internally) for resolving it)
2. Assign lib/cli/execution-span.js promise to serverless.executionSpan (as it can be valuable for a plugins). Remove it's so far counterpart serverless.onExitPromise and in places it was used, refer to either promise exported by lib/cli/execution-span.js or serverless.executionSpan
3. Move analytics.sendPending() to top of the try/catch clause

1.0 Seclude -v, --version CLI params handling

1. At begin of the flow implement following:
   • If CLI command is sls -v [...] or sls --version [...] Generate simple version output which follows one implemented internally in CLI class. Place handling logic in lib/cli/eventually-output-version-info.js (and ensure some tests for it)
   • Abort the process (ensure no further process steps are pursued (but without hard exiting the process))
2. Remove -v, --version option handling and recognition from CLI.js class

2.0 Resolve eventual service config file path (service context)

1. Follow up with service context detection logic. It should be implemented in lib/cli/resolve-service-config-path.js and resemble logic we have now here:

   serverless/lib/utils/getServerlessConfigFile.js

   Lines 13 to 48 in bde334c

   	const getConfigFilePath = async (servicePath, options = {}) => {
   	if (options.config) {
   	const customPath = path.join(servicePath, options.config);
   	return fileExists(customPath).then(exists => {
   	return exists ? customPath : null;
   	});
   	}
   	const jsonPath = path.join(servicePath, 'serverless.json');
   	const ymlPath = path.join(servicePath, 'serverless.yml');
   	const yamlPath = path.join(servicePath, 'serverless.yaml');
   	const jsPath = path.join(servicePath, 'serverless.js');
   	const tsPath = path.join(servicePath, 'serverless.ts');
   	const [jsonExists, ymlExists, yamlExists, jsExists, tsExists] = await Promise.all([
   	fileExists(jsonPath),
   	fileExists(ymlPath),
   	fileExists(yamlPath),
   	fileExists(jsPath),
   	fileExists(tsPath),
   	]);
   	if (yamlExists) {
   	return yamlPath;
   	} else if (ymlExists) {
   	return ymlPath;
   	} else if (jsonExists) {
   	return jsonPath;
   	} else if (jsExists) {
   	return jsPath;
   	} else if (tsExists) {
   	return tsPath;
   	}
   	return null;
   	};

2. Remove from internals any config file path resolution logic and ensure that above is handled as single point of truth:
   • Pass resolved config file path to Serverless constructor on config.serviceConfigPath and assign it to serviceConfigPath property
   • Resolve config.servicePath from serviceConfigPath
   • Remove findServicePath util
   • Replace getConfigFilePath usage with resolveServiceConfigPath at interactive CLI setup
   • Replace any getServerlessConfigFilePath usage with serverless.serviceConfigPath and remove this util entirely (in interactive CLI setup simply override serverless.serviceConfigPath instead)
   • Remove getConfigFilePath

2.1 Parse service config file source

1. Follow up with service config content resolution (bare resolution with no normalization or vars resolution at this point). It should be implemented in lib/service-config/read-source.js, and resemble logic we have here:

   serverless/lib/utils/getServerlessConfigFile.js

   Lines 112 to 127 in bde334c

   	getServerlessConfigFilePath(serverless).then(configFilePath => {
   	if (!configFilePath) return null;
   	const fileExtension = path.extname(configFilePath);
   	const isJSOrTsConfigFile = fileExtension === '.js' || fileExtension === '.ts';
   	return (isJSOrTsConfigFile
   	? handleJsOrTsConfigFile(configFilePath)
   	: readFile(configFilePath)
   	).then(config => {
   	if (_.isPlainObject(config)) return config;
   	throw new ServerlessError(
   	`${path.basename(configFilePath)} must export plain object`,
   	'INVALID_CONFIG_OBJECT_TYPE'
   	);
   	});
   	})

   . if readServiceConfigSource crashes expose the error only if it's not CLI help request, otherwise behave as we're not in service context.
2. Remove from internals any config source resolution:
   • Pass resolved config source to Serverless constructor on config.serviceConfigSource and assign it to serviceConfigSource property
   • In Servervless.js class methods, replace this.pluginManager.serverlessConfigFile references with this.serviceConfigSource
   • In interactive CLI setup override serverless.serviceConfigSource with help of readServiceConfigSource module.
   • Remove pluginManager.loadConfigFile method and pluginManager.serverlessConfigFile property

2.2 Initial (partial) variables resolution

For that we would need to Implement new variables resolver with following modules:

lib/variables/resolve-variables-map.js

Function that takes serviceConfig as an input. Traverses it's properties and returns map of all properties which use variable syntax. Result map should expose all information needed for complete variables resolution without a need of repeated property parsing.

After we will fully override variable resolution that's currently in a framework (point 5.2), function should be configured to also override all serviceConfig properties which depend on variables with null values (technically we move all information to variables map, and remove it from serviceConfig. It's to ensure that eventual further serviceConfig processing in case of variable resolution errors is not affected by unresolved properties content)

Expected format of result map

const sep = "\0";

const exampleResultMap = {
  [`custom${sep}creds`]: {
    raw:
      '${file(../config.${opt:stage, self:provider.stage, "dev"}.json):CREDS}_${self:custom.foo}',
    meta: [
      // Start from last to first
      // If vars are part of a string, provide start and end locations
      // and unconditionally coerce result to string
      { start: 71, end: 89, sources: [{ source: 'self', address: { raw: 'custom.foo' } }] },
      {
        start: 0,
        end: 70,
        sources: [
          {
            source: 'file',
            param: {
              raw: '../config.${opt:stage, self:provider.stage, "dev"}.json',
              meta: [
                {
                  start: 10,
                  end: 50,
                  sources: [
                    { source: 'opt', address: { raw: 'stage' } },
                    { source: 'self', address: { raw: 'provider.stage' } },
                    { raw: 'dev' },
                  ],
                },
              ],
            },
            address: { raw: 'CREDS' },
          },
        ],
      },
    ],
  },
  [`layers${sep}hello${sep}path`]: {
    raw: '${self:custom.layerPath}',
    // If property value is entirely constructed with var
    // No start/end points need to be configured
    // In such case we also support any result type (no string coercion)
    variables: [{ sources: [{ source: 'self', address: { raw: 'custom.layerPath' } }] }],
  },
};

Note: In case of resolution from external files, new added content will need to have eventual variables resolved through same util

lib/variables/resolve-variables.js

Function that takes serviceConfig, variablesMap and variablesResolvers as an input.

variablesResolvers is expected to be a simple map with source type as a key (e.g. self, fileetc.) and function that takesserviceConfig` and eeventual param configured for resolver as arguments. Function may return result sync way or async via returned promise

There should be attempt to resolve every property.

• If resolution succeed, resolved value should be assigned on serviceConfig object and variable reference removed from variablesMap.
• if resolution for any would fail it should be reported by an error which trough code resembles why it failed (A. not supported source, B. invalid configuration, C. missing necessary input, D. external service error). Failed resolution attempt should be stored in variablesMap (in future processing, resolution should be reattempted only if fail was caused by A error, in other cases there should be no retry.

If there's any fail. Function crashes, and on it's error it should expose errorneusVariableKeys property with keys to each variable resolutions that failed.

Having above:

1. Generate variables map
2. Attempt to resolve file, self and strToBool variable sources (but only those not depending on other resolvers). If it fails ignore any not supported source errors. If there are other errors and it's not a CLI help request, in initial stage, ignore them, but after addressing 5.2 signal them with warning message and show a deprecation that with next major we will fail.

2.3 Ensure provider and provider.stage properties are resolved.

1. Inspect variables map:
   • if provider property still depends on variable resolution, crash with meaningful error, that we cannot accept given form of configuration
   • if provider.stage property still depends on variable resolution. Show warning and deprecation, stating that it's not recommend to use variables at this property and that we will fail on that with next major

2.4 Ensure to load env variables from .env files

Follow up with resolution of environment variables from .env files (currently being implemented at #8413)

2.5 Further (partial) variables resolution

As in 2.1 step, attempt to resolve file, self, strToBool and env variable sources (but only those not depending on other resolvers). If it fails ignore any not supported source errors. If there are other errors and it's not a CLI help request in initial stage, ignore them, but after addressing 5.2 signal them with warning message and show a deprecation that with next major we will fail.

2.6 Ensure eventual plugins property is fully resolved.

Inspect variables map, if plugins property still depends on variable resolution, crash with meaningful error, that we cannot accept given form of configuration

2.7.0 Recognize help command

1. Implement is help CLI command logic in lib/cli/is-help-command.js (it should follow cli.isHelpRequest logic but also recognize --help-components) and adapt it in internals:
   • Resolve isHelpCommand and pass it to Serverless constructor in config.isHelpCommand and internally assign it to isHelpCommand property
   • Replace internal serverless.cli.isHelpRequest usage with serverless.isHelpCommand
   • Replace internal pluginManager.cliOptions.help usage with serverless.isHelpCommand
   • Remove serverless.cli.isHelpRequest implementation

2.7.1 Recognize commands which are independent of external plugins

Handling of those commands ideally should be totally secluded from Framework engine, still to not impose too timetaking refactor at this step let's simply mark them, to make further processing possible (having that addressed, let's open an issue calling for their seclusion)

1. If CLI command is either plugin, login, logout or dashboard pass to Serverless constructor a shouldMutePluginInitializationErrors: true option, and internally assign t to _shouldMutePluginInitializationErrors property
2. In pluginManager.resolveServicePlugins() Rely on serverles._shouldMutePluginInitializationErrors and remove pluginManager.pluginIndependentCommands property.

2.7.2 Initialize Serverless instance

(this will most likely lay out naturally and should not require any code changes)

Follow up with construction of Serverless instance and invoke of serverless.init()

3.0 If CLI help command show help and abort

1. Implement display option help logic (to be used by various help variants) in lib/cli/help/options.js. It should take our common command configuration object, and resemble logic we have at cli.displayCommandOptions()
2. Implement interactive CLI help logic in lib/cli/help/interactive.js. It should be a function that accepts an interactiveCLI command configuration and resembles logic we have at `cli.generateInteactiveCliHelp()
3. Implement main CLI help logic in lib/cli/help/framework.js. It should be a function that accepts a loadedPlugins and resembles logic we have at cli.generateMainHelp() (note we should have CLI: Remove help --verbose option and improve general help output #8497 addressed at this point)
4. Implement specific command help logic in lib/cli/help/command.js. It should be a function that accepts commandName and command arguments, and:
   • Dislpays plugin name if command is configured via external plugin
   • Displays command usage and it's options - Note I believe we can implement inline cli.displayCommandUsage() logic, and refer to already implemented lib/cli/help/options.js
5. if interactive CLI help request. Find InteractiveCli plugin, run lib/cli/help/interactive.js with its comand and abort
6. If general (not command specific) help request, run lib/cli/help/framework.js with serverless.cli.loadedCommands
7. If command help request, find given command in serverless.cli.loadedCommands
   1. If specified command is not found show warning and output general help
   2. Otherwise run lib/cli/help/command.js with resolved comman
8. Remove following code:
   • cli.displayHelp call
   • All help generation commands from CLI class

4.0 Parse CLI arguments

1. Having a map of all supported commands and options follow up with resolution of CLI arguments:
   • Parse CLI args with logic as in resolveCliInput
   • Ensure that all CLI args validation steps (aside of validateServerlessConfigDependency and assignDefaultOptions) as pursued in pluginManager.invoke are taken care of. Ideally if it's generalized, so can be also used to validate Components CLI input
   • Let's put it into lib/cli/parse-params.js
2. Pass resolved commands and options to serverless.run() method. In context serverless.run() assign those properties onprocessedInput property
   • To not break things for external plugins we cannot just remove resolution of processedInput, that happens in serverless.init(). Still let's override there processedInput with getter that exposes a deprecation message if property is accessed at initialization phase (having that we will remove it next major)
   • In each internal plugin remove handling of second constructor option (CLI options) - (it'll also automatically address other important concern -> Ensure options as passed to plugins is not modified #2582). If for some reason reliance on CLI options seems crucial at implementation phase, then move its handling to initialize lifecycle hook (it's first lifecycle event propagated unconditionally). Access CLI options from serverless.processedInput (and treat it as read only)
   • Refactor pluginManager.validateOptions so it's eventual errors do not refer to CLI params (this method will now be effective only for programmatic usage)
   • Refactor pluginManager.validateServerlessConfigDependency so it's eventual errors do not refer to CLI usage (e.g. we should refer to service context and not to service directory)
   • Remove pluginManger.convertShortcutsIntoOptions as Framework will already be populated with resolved shortcuts

5.1 Resolve variables for all variable sources which do not depend on config properties

As in 2.1 step, attempt to resolve all variable sources which do not depend on config properties.

If it fails ignore any not supported source errors. If there are other errors in initial stage, ignore them, but after addressing 5.2 signal them with warning message and show a deprecation that with next major we will fail.

5.2 Resolve all remaining variables in service config

As in 2.1 step, attempt to resolve all remaining variables.

If it fails signal them with warning message and show a deprecation that with next major we will fail. Additionally:

• Ensure that after resolving variables map (point 2.2) all service config properties configure through variables are preset to null
• In any variable resolution step, convert errors ignoring to warnings with and deprecation that with next major we will fail.

Remove all variable resolution logic from Framework core

6.0 Run lifecycle events for CLI command

(this will most likely lay out naturally and should not require any code changes)

Follow up with serverless.run()

Progress summary:

• 0.0.1 - Tests refactor: Move all unit tests to "test/unit" folder #8478 - Move all unit tests to test/unit folder
• 0.0.2 - CLI: Remove help --verbose option and improve general help output #8497 - Remove CLI help --verbose option and improve general help output
• 0.1.0 - Unify process error handling
• 0.2.0 - Generalize process execution span promise handling
• 1.0.0 - Seclude -v, --version CLI params handling
• 2.0.0 - Resolve eventual service config file path (service context)
• 2.1.0 - Parse service config file source
• 2.2.0 - Initial (partial) variables resolution
• 2.3.0 - Ensure provider and provider.stage properties are resolved.
• 2.4.0 - Ensure to load env variables from .env files
• 2.5.0 - Further (partial) variables resolution
• 2.6.0 - Ensure eventual plugins property is fully resolved
• 2.7.0 - Recognize help command
• 2.7.1 - Recognize commands which are independent of external plugins
• 2.7.2 - Initialize Serverless instance
• 3.0.0 - If CLI help command show help and abort
• 4.0.0 - Parse CLI arguments
• 5.1.0 - Resolve variables for all variable sources which do not depend on config properties
• 5.2.0 - Resolve all remaining variables in service config
• 6.0.0 - Run lifecycle events for CLI command