tella autodocs

Alphabetical index

tella.agents module

tella.curriculum module

Copyright © 2021-2022 The Johns Hopkins University Applied Physics Laboratory LLC

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

exception tella.curriculum.ValidationError

Raised when there is a problem with a curriculum.

class tella.curriculum.TaskVariant(task_cls: Type[Env], *, rng_seed: int, params: Dict | None = None, task_label: str | None = None, variant_label: str | None = 'Default', num_episodes: int | None = None, num_steps: int | None = None)

A TaskVariant represents a fixed number of steps or episodes in a single type of gym.Env.

__init__(task_cls: Type[Env], *, rng_seed: int, params: Dict | None = None, task_label: str | None = None, variant_label: str | None = 'Default', num_episodes: int | None = None, num_steps: int | None = None) → None
Parameters:

  • task_cls – the gym.Env of this task variant

  • rng_seed – An integer seed used to repeatably instantiate the environment

  • params – An optional dict of parameters to be passed as constructor arguments to the environment.

  • task_label – The name of the task which describes this environment.

  • variant_label – The name of the variant (of task_label) which describes this environment.

  • num_episodes – The length limit of this experience in number of episodes.

  • num_steps – The length limit of this experience in number of steps.

Raises:
validate() → None

A method to validate that the experience is set up properly.

Raises:

ValidationError – if the experience is not set up properly.

make_env() → Env

Initializes the gym environment object

class tella.curriculum.TaskBlock(task_label: str, task_variants: Iterable[TaskVariant])

A TaskBlock represents a sequence of one or more TaskVariant which all share the same general task.

__init__(task_label: str, task_variants: Iterable[TaskVariant]) → None
Parameters:

  • task_label – The name of the task which describes the environment for all contained variants.

  • task_variants – A sequence of one or more TaskVariant

task_variants() → Iterable[TaskVariant]
Returns:

An Iterable of TaskVariant.

class tella.curriculum.Block(task_blocks: Iterable[TaskBlock])

Represents a sequence of one or more TaskBlock

abstract property is_learning_allowed: bool
Returns:

Bool indicating if this block is intended for learning or evaluation

__init__(task_blocks: Iterable[TaskBlock]) → None
Parameters:

task_blocks – A sequence of one or more TaskBlock

task_blocks() → Iterable[TaskBlock]
Returns:

An Iterable of TaskBlock.

class tella.curriculum.LearnBlock(task_blocks: Iterable[TaskBlock])

A Block where the data can be used for learning.

class tella.curriculum.EvalBlock(task_blocks: Iterable[TaskBlock])

A Block where the data can NOT be used for learning.

class tella.curriculum.AbstractCurriculum(rng_seed: int, config_file: str | None = None)

Represents a lifelong/continual learning curriculum. A curriculum is simply a sequence of one or more Block.

__init__(rng_seed: int, config_file: str | None = None)
Parameters:

  • rng_seed – The seed to be used in setting random number generators.

  • config_file – Path to a config file for the curriculum or None if no config.

copy() → AbstractCurriculum
Returns:

A new instance of this curriculum, initialized with the same arguments.

Note

Curriculum authors will need to overwrite this method for subclasses with different arguments.

abstract learn_blocks_and_eval_blocks() → Iterable[Block]

Generate the learning and eval blocks of this curriculum.

Returns:

An Iterable of Block.

validate() → None

A method to validate that the curriculum is set up properly. This is an optional capability for curriculum authors to implement.

Raises:

ValidationError – if the curriculum is not set up properly.

class tella.curriculum.InterleavedEvalCurriculum(rng_seed: int, config_file: str | None = None)

A common curriculum format where a single evaluation block is repeated before, between, and after the curriculum’s learning blocks.

This class implements AbstractCurriculum.learn_blocks_and_eval_blocks(), and expects the user to implement two new methods:

  1. learn_blocks(), which returns the sequence of LearnBlock.

  2. eval_block(), which returns the single EvalBlock to be interleaved between each LearnBlock.

__init__(rng_seed: int, config_file: str | None = None)
Parameters:

  • rng_seed – The seed to be used in setting random number generators.

  • config_file – Path to a config file for the curriculum or None if no config.

abstract learn_blocks() → Iterable[LearnBlock]
Returns:

An iterable of LearnBlock.

abstract eval_block() → EvalBlock
Returns:

The single EvalBlock to interleave between each individual LearnBlock returned from InterleavedEvalCurriculum.learn_blocks().

learn_blocks_and_eval_blocks() → Iterable[Block]

Generate the learning and eval blocks of this curriculum.

Returns:

An Iterable of Block.

tella.curriculum.split_task_variants(task_variants: Iterable[TaskVariant]) → Iterable[TaskBlock]

Divides task variants into one or more blocks of matching tasks

Parameters:

task_variants – The iterable of TaskVariant to be placed into task blocks.

Returns:

An iterable of one or more TaskBlock which contain the provided task variants.

tella.curriculum.simple_learn_block(task_variants: Iterable[TaskVariant]) → LearnBlock

Constructs a LearnBlock with the task variants passed in. TaskBlock are divided as needed.

Parameters:

task_variants – The iterable of TaskVariant to include in the LearnBlock.

Returns:

A LearnBlock with one or more TaskBlock which contain the provided task variants.

tella.curriculum.simple_eval_block(task_variants: Iterable[TaskVariant]) → EvalBlock

Constructs a EvalBlock with the task variants passed in. TaskBlock are divided as needed.

Parameters:

task_variants – The iterable of TaskVariant to include in the EvalBlock.

Returns:

A EvalBlock with one or more TaskBlock which contain the provided task variants.

class tella.curriculum.Observation

Observation of environment state

alias of TypeVar(‘Observation’)

class tella.curriculum.Action

Action taken in environment

alias of TypeVar(‘Action’)

tella.curriculum.Reward

Float reward from environment

tella.curriculum.Done

Bool, True if episode has ended

class tella.curriculum.Transition(observation: Observation, action: Action, reward: float | None, done: bool, next_observation: Observation)

A named tuple containing data from a single step in an MDP: (observation, action, reward, done, next_observation)

observation: Observation

Alias for field number 0

action: Action

Alias for field number 1

reward: float | None

Alias for field number 2

done: bool

Alias for field number 3

next_observation: Observation

Alias for field number 4

tella.curriculum.summarize_curriculum(curriculum: AbstractCurriculum) → str

Generate a detailed string summarizing the contents of the curriculum.

Returns:

A string that would print as a formatted outline of this curriculum’s contents.

tella.curriculum.validate_curriculum(curriculum: AbstractCurriculum)

Helper function to do a partial check that task variants are specified correctly in the blocks of the AbstractCurriculum.

Uses AbstractCurriculum.validate() if implemented for the curriculum. Uses TaskVariant.validate() to check task variants.

Raises:
Returns:

None

tella.curriculum.validate_params(fn: Callable, param_names: List[str]) → None

Determines whether there are missing or invalid names in param_names to pass to the function fn.

Warning

if fn has any **kwargs, then all arguments are valid and this method won’t be able to verify anything.

Parameters:

  • fn – The callable that will accept the parameters.

  • param_names – The names of the parameters to check.

Raises:

  • ValidationError – if any of param_names are not found in the signature, and there are no **kwargs

  • ValidationError – if any of the parameters without defaults in fn are not present in param_names

  • ValidationError – if any *args are found

  • ValidationError – if any positional only arguments are found (i.e. using /)

tella.experiment module

Copyright © 2021-2022 The Johns Hopkins University Applied Physics Laboratory LLC

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

tella.experiment.AgentFactory

AgentFactory is a function or class that returns a ContinualRLAgent.

It takes 5 arguments, which are the same as ContinualRLAgent.__init__():

  1. rng_seed, which is an integer to be used for repeatable random number generation

  2. observation_space, which is a gym.Space

  3. action_space, which is a gym.Space

  4. num_parallel_envs, which is an integer indicating how many environments will be run in parallel at the same time.

  5. config_file, which is a path as a string to a configuration file or None if no configuration.

A concrete subclass of ContinualRLAgent can be used as an AgentFactory:

class MyAgent(ContinualRLAgent):
    ...

agent_factory: AgentFactory = MyAgent
agent = agent_factory(rng_seed, observation_space, action_space, num_parallel_envs, config_file)

A function can also be used as an AgentFactory:

def my_factory(rng_seed, observation_space, action_space, num_parallel_envs, config_file):
    ...
    return my_agent(rng_seed, observation_space, action_space, num_parallel_envs, config_file)

agent_factory: AgentFactory = my_factory
agent = agent_factory(rng_seed, observation_space, action_space, num_parallel_envs, config_file)

alias of Callable[[int, Space, Space, int, str], ContinualRLAgent]

tella.experiment.CurriculumFactory

CurriculumFactory is a function or class that returns an AbstractCurriculum.

It takes 2 arguments:

  1. an integer which is to be used for repeatable random number generation

  2. an option filepath to be loaded as a configuration dict.

A concrete subclass of AbstractCurriculum can be used as an CurriculumFactory:

class MyCurriculum(AbstractCurriculum[AbstractRLTaskVariant]):
    ...

curriculum_factory: CurriculumFactory = MyCurriculum
curriculum = curriculum_factory(rng_seed, config_file)

A function can also be used as a CurriculumFactory:

def my_factory(rng_seed, config_file):
    ...
    return my_curriculum(rng_seed, config_file)

curriculum_factory: CurriculumFactory = my_factory
curriculum = curriculum_factory(rng_seed, config_file)

alias of Callable[[int, Optional[str]], AbstractCurriculum]

tella.experiment.rl_experiment(agent_factory: Callable[[int, Space, Space, int, str], ContinualRLAgent], curriculum_factory: Callable[[int, str | None], AbstractCurriculum], num_lifetimes: int, num_parallel_envs: int, log_dir: str, agent_seed: int | None = None, curriculum_seed: int | None = None, render: bool | None = False, agent_config: str | None = None, curriculum_config: str | None = None, lifetime_idx: int = 0) → None

Run an experiment with an RL agent and an RL curriculum.

Parameters:

  • agent_factory – Function or class to produce agents.

  • curriculum_factory – Function or class to produce curriculum.

  • num_lifetimes – Number of times to call run().

  • num_parallel_envs – Number of parallel environments.

  • log_dir – The root log directory for l2logger.

  • lifetime_idx – The index of the lifetime to start running with. This will skip the first N seeds of the RNGs, where N = lifetime_idx.

  • agent_seed – The seed for the RNG for the agent or None for random seed.

  • curriculum_seed – The seed for the RNG for the curriculum or None for random seed.

  • render – Whether to render the environment for debugging or demonstrations.

  • agent_config – Optional path to a configuration file for the agent.

  • curriculum_config – Optional path to a configuration file for the curriculum.

Returns:

None

tella.experiment.run(agent: ContinualRLAgent, curriculum: AbstractCurriculum, render: bool | None, log_dir: str, num_envs: int | None = 1)

Run an agent through an entire curriculum.

Parameters:

  • agent – Agent for this experiment.

  • curriculum – Curriculum for this experiment.

  • render – Bool flag to toggle environment rendering.

  • log_dir – Directory for l2logger files.

  • num_envs – Number of parallel environments.

class tella.experiment.L2Logger(log_dir: str, scenario_dir: str, scenario_info: Dict[str, str], num_envs: int)

A utility class for handling logging with the l2logger package.

The l2logger package logs cumulative episode rewards when episodes finish, and all of this can be achieved by tracking data from transitions via the L2Logger.receive_transitions().

Additionally, l2logger needs to track the global block number and task id information, so L2Logger.block_start() and L2Logger.task_variant_start() should be called at the appropriate times to track this information.

__init__(log_dir: str, scenario_dir: str, scenario_info: Dict[str, str], num_envs: int) → None
tella.experiment.ActionFn

A function that takes a list of Observations and returns a list of Actions, one for each observation.

alias of Callable[[List[Optional[Observation]]], List[Optional[Action]]]

tella.experiment.generate_transitions(task_variant: TaskVariant, action_fn: Callable[[List[Observation | None]], List[Action | None]], num_envs: int, render: bool = False) → Iterable[List[Transition | None]]

Yields markov transitions from the interaction between the action_fn and the gym.Env contained in TaskVariant.

Note

None transitions

Extra data can be produced when using num_envs > 1, if the data limits in TaskVariant % num_envs != 0. For an example if the limit is 4 episodes, and num_envs is 5, then this function will generate a whole extra episode worth of transitions. In order to prevent the leak of extra data, we mask out any transitions above the data limit by setting them to None.

Parameters:

  • task_variant – The task variant containing environment and seed information.

  • action_fn – Selects actions to take in the environment given an observation.

  • num_envs – Controls the amount of parallelization to use for this episodic task variant. See gym.vector.VectorEnv for more information.

  • render – Whether to render the environment at each step.

Returns:

A generator of transitions.

tella.experiment.hide_rewards(transitions: List[Transition | None]) → List[Transition | None]

Masks out any rewards in the transitions passed in by setting the reward field to None.

Parameters:

transitions – The transitions to hide the reward in

Returns:

The new list of transitions with all rewards set to None

tella.cli module

Copyright © 2021-2022 The Johns Hopkins University Applied Physics Laboratory LLC

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

tella.cli.rl_cli(agent_factory: Callable[[int, Space, Space, int, str], ContinualRLAgent], curriculum_factory: Callable[[int, str | None], AbstractCurriculum] | None = None) → None

Builds a CLI wrapper around tella.experiment.rl_experiment() to enable running experiments with the agent produced by the provided AgentFactory. The experiment’s curriculum is either provided here or else loaded as a command line argument.

Example:

import tella

class MyAgent(ContinualRLAgent):
    ...

if __name__ == "__main__":
    tella.rl_cli(MyAgent)
Parameters:

  • agent_factory – A function or class producing a ContinualRLAgent.

  • curriculum_factory – Optional curriculum factory producing an AbstractCurriculum to support running experiments with a fixed curriculum. Otherwise, the curriculum is specified on the command line.