cambrian.eyes¶

This module defines the Cambrian eyes.

Submodules¶

Classes¶

`MjCambrianEye`	Defines an eye for the cambrian environment. It essentially wraps a mujoco Camera
`MjCambrianEyeConfig`	Defines the config for an eye. Used for type hinting.
`MjCambrianMultiEye`	Defines a multi-eye system that procedurally generates multiple eyes and manages
`MjCambrianMultiEyeConfig`	Config for MjCambrianMultiEye.
`MjCambrianOpticsEye`	This class applies the depth invariant PSF to the image.
`MjCambrianOpticsEyeConfig`	This defines the config for the optics module. This extends the base eye config

Package Contents¶

class MjCambrianEye(config, name, *, disable_render=False)[source]¶

Defines an eye for the cambrian environment. It essentially wraps a mujoco Camera object and provides some helper methods for rendering and generating the XML. The eye is attached to the parent body such that movement of the parent body will move the eye.

Parameters:

config (MjCambrianEyeConfig) – The configuration for the eye.
name (str) – The name of the eye.

Keyword Arguments:

disable_render (bool) – Whether to disable rendering. Defaults to False. This is useful for derived classes which don’t intend to use the default rendering mechanism.

generate_xml(parent_xml, geom, parent_body_name=None, parent=None)[source]¶

Generate the xml for the eye.

In order to combine the xml for an eye with the xml for the agent that it’s attached to, we need to replicate the path with which we want to attach the eye. For instance, if the body with which we want to attach the eye to is at mujoco/worldbody/torso, then we need to replicate that path in the new xml. This is kind of difficult with the xml library, but we’ll utilize the CambrianXML helpers for this.

Parameters:

parent_xml (MjCambrianXML) – The xml of the parent body. Used as a reference to extract the path of the parent body.
geom (MjCambrianGeometry) – The geometry of the parent body. Used to calculate the pos and quat of the eye.
parent_body_name (Optional[str]) – The name of the parent body. Will search for the body tag with this name, i.e. <body name=”<parent_body_name>” …>. Either this or parent must be set.
parent (Optional[List[Element] | Element]) – The parent element to attach the eye to. If set, parent_body_name will be ignored. Either this or parent_body_name must be set.

reset(spec)[source]¶: Sets up the camera for rendering. This should be called before rendering the first time.

step(obs=None)[source]¶

Simply calls render and sets the last observation. See render() for more information.

Parameters:: obs (Optional[ObsType]) – The observation to set. Defaults to None. This can be used by derived classes to set the observation directly.

render()[source]¶

Render the image from the camera. Will always only return the rgb array.

This differs from step in that this is a debug method. The rendered image here will be used to visualize the eye in the viewer.

property config: MjCambrianEyeConfig¶: The config for the eye.

property name: str¶: The name of the eye.

property observation_space: gymnasium.spaces.Box¶: Constructs the observation space for the eye. The observation space is a spaces.Box with the shape of the resolution of the eye.

property prev_obs: torch.Tensor¶: The last observation returned by self.render().

class MjCambrianEyeConfig[source]¶

Bases: hydra_config.HydraContainerConfig

Defines the config for an eye. Used for type hinting.

Variables:

instance (Callable[[Self, str], MjCambrianEye]) – The class instance to use when creating the eye. Takes the config and the name of the eye as arguments.
fov (Tuple[float, float]) – Independent of the fovy field in the MJCF xml. Used to calculate the sensorsize field. Specified in degrees. Mutually exclusive with fovy. If focal is unset, it is set to 1, 1. Will override sensorsize, if set. Fmt: fovy fovx.
focal (Tuple[float, float]) – The focal length of the camera. Fmt: focal_y focal_x.
sensorsize (Tuple[float, float]) – The size of the sensor. Fmt: height width.
resolution (Tuple[int, int]) – The width and height of the rendered image. Fmt: height width.
coord (Tuple[float, float]) – The x and y coordinates of the eye. This is used to determine the placement of the eye on the agent. Specified in degrees. This attr isn’t actually used by eye, but by the agent. The eye has no knowledge of the geometry it’s trying to be placed on. Fmt: lat lon
orthographic (bool) – Whether the camera is orthographic
renderer (MjCambrianRendererConfig) – The renderer config to use for the underlying renderer.

class MjCambrianMultiEye(config, name, disable_render=True)[source]¶

Bases: cambrian.eyes.eye.MjCambrianEye

Defines a multi-eye system that procedurally generates multiple eyes and manages them.

Inherits from MjCambrianEye but manages multiple eyes.

Parameters:

config (MjCambrianMultiEyeConfig) – Configuration for the multi-eye system.
name (str) – Base name for the eyes.

generate_xml(parent_xml, geom, parent_body_name)[source]¶: Generate the XML for all eyes.

reset(*args)[source]¶: Reset all eyes.

step(obs=None)[source]¶: Step all eyes and collect observations.

render()[source]¶

This is a debug method which renders the eye’s as a composite image.

Will appear as a compound eye. For example, if we have a 3x3 grid of eyes:: TL T TR ML M MR BL B BR

Each eye has a red border around it.

property observation_space: gymnasium.spaces.Space¶: Constructs the observation space for the multi-eye.

property prev_obs: cambrian.utils.types.ObsType¶: The last observations from all eyes.

property eyes: Dict[str, cambrian.eyes.eye.MjCambrianEye]¶: Returns the dictionary of eyes.

property name: str¶: Returns the base name of the multi-eye system.

property num_eyes: int¶: Returns the number of eyes.

class MjCambrianMultiEyeConfig[source]¶

Bases: cambrian.eyes.eye.MjCambrianEyeConfig

Config for MjCambrianMultiEye.

Inherits from MjCambrianEyeConfig and adds attributes for procedural eye placement.

Variables:

instance (Callable[[Self, str], MjCambrianEye]) – The class instance to use when creating the eye. Takes the config and the name of the eye as arguments.
single_eye (MjCambrianEyeConfig) – The config for a single eye. This is used as the base configuration for all eyes in the multi-eye system.
lat_range (Optional[Tuple[float, float]]) – The x range of the eye. This is used to determine the placement of the eye on the agent. Specified in degrees. This is the latitudinal/vertical range of the evenly placed eye about the agent’s bounding sphere.
lon_range (Optional[Tuple[float, float]]) – The y range of the eye. This is used to determine the placement of the eye on the agent. Specified in degrees. This is the longitudinal/horizontal range of the evenly placed eye about the agent’s bounding sphere.
num_eyes (Optional[Tuple[int, int]]) – The num of eyes to generate. If this is specified, then the eyes will be generated on a spherical grid. The first element is the number of eyes to generate latitudinally and the second element is the number of eyes to generate longitudinally. The eyes will be named sequentially starting from eye_0. Each eye will default to use the first eye config in the eyes attribute. eyes must have a length of 1 if this is specified. Each eye is named eye_{lat}_{lon} where lat is the latitude index and lon is the longitude index.
flatten_observation (Optional[bool]) – Whether to flatten the observation space of the multi-eye system. If True, the observation space will be a Box space with the shape (num_eyes * eye_observation_space,). If False, the observation space will be a Dict space with the keys as the eye names and the values as the eye observation spaces. Defaults to False.

class MjCambrianOpticsEye(config, name)[source]¶

Bases: cambrian.eyes.eye.MjCambrianEye

This class applies the depth invariant PSF to the image.

Parameters:: config (MjCambrianOpticsConfig) – Config for the optics module.

initialize()[source]¶: This will initialize the parameters used during the PSF calculation.

step(obs=None)[source]¶: Overwrites the default render method to apply the depth invariant PSF to the image.

class MjCambrianOpticsEyeConfig[source]¶

Bases: cambrian.eyes.eye.MjCambrianEyeConfig

This defines the config for the optics module. This extends the base eye config and adds additional parameters for the optics module.

Variables:

pupil_resolution (Tuple[int, int]) – Resolution of the pupil plane. This is used to calculate the PSF.
noise_std (float) – Standard deviation of the Gaussian noise to be added to the image. If 0.0, no noise is added.
wavelengths (Tuple[float, float, float]) – Wavelengths of the RGB channels.
f_stop (float) – F-stop of the lens. This is used to calculate the PSF.
refractive_index (float) – Refractive index of the lens material.
height_map (List[float]) – Height map of the lens. This is used to calculate the phase shift of the light passing through the lens. Uses a radially symmetric approximation.
scale_intensity (bool) – Whether to scale the intensity of the PSF by the overall throughput of the aperture.
aperture (MjCambrianApertureConfig) – Aperture config. This defines the aperture of the lens. The aperture can be circular or custom.
depths (List[float]) – Depths at which the PSF is calculated. If empty, the psf is calculated for each render call; otherwise, the PSFs are precomputed.