SimPEG 0.14.0 Release Notes

This release marks a major change in the structure of SimPEG, and it all started with this,

  • #562: What’s the problem with Problem?

We hope to answer that question with this release!

This release will break backward compatibility as many modules and classes have been renamed. Check out the Gallery and Tutorials (which have also been improved) to see how it’s done.

We are also only supporting Python versions >=3.6 at this time. We have dropped all testing and development on Python versions previous to this (especially 2.7).

Highlights

  • No more Problem-s, only Simulation-s

  • Data is important

  • PEP 8 renaming

  • Dask parallelism

  • Resistivity Simulation improvements

Simulation

We will refer to this update as the simulation update, and there are a few large changes that we will attempt to describe here.

We (the developers) believed that there were some significant challenges with the overall structure of the SimPEG framework, which all revolved around constructing a forward simulation by pair-ing a Survey and a Problem. The Survey handled things like sources, receivers, and the Problem handled the physics engine of the forward simulation. These two items had to be created separately, then pair-ed afterwards for both the Problem to be able to use sources, and for the Survey to predict data. We found that this structure made it difficult to interface with other packages and was also generally difficult to explain. Also, field data was then attached to the Survey class.

These changes within this section are also the ones which will require the most changes to code (which are still hopefully small).

The Simulation class

Problem has been renamed to Simulation

We decided to refactor this code into something a little more understandable. The Simulation class is now the workhorse of the SimPEG forward simulations. It handles things like modeling fields and projecting those fields to the data locations defined by its survey. The Survey class is much lighter weight, now only handling the sources and receivers. Also a single survey can now be attached to many Simulation-s.

Previously we had something like,

survey = DC.Survey(srcList)
prob = DC.Problem3D_CC(mesh, rhoMap=mapping)
prob.pair(survey)

# Compute the fields from `prob`
fields = prob.fields(model)
# And predict data using the `survey`
dpred = survey.dpred(model, f=fields)

Now,

survey = resistivity.Survey([source_list])
sim = resistivity.Simulation3DCellCentered(
    mesh,
    survey=survey,
    rhoMap=mapping
)

# Compute the fields from `sim`
fields = sim.fields(model)
# Predict data also using `sim`
dpred = sim.dpred(model, f=fields)

See? The Simulation handles predicting data. This change will also make it easier to interface with external codes for inversion purposes, as all that is needed to be defined to use a Simulation for an InvProblem, is sim.dpred, sim.Jvec and sim.Jtvec.

Please see the documentation for the SimPEG.simulation.BaseSimulation class as well as the individual methods’ Simulation-s, for a detailed description of arguments, but largely it accepts the same arguments as the Problem class, but now also requires a Survey to be set.

The Data class

Previously, field data would also live within the Survey class. Which was not only confusing, but placed the importance on the wrong component. When inverting geophysical data, we are concerned with the data. Thus we would like to enforce this importance by making data live in a dedicated Data class. This Data class can act like a smart dictionary to grab data associated with a specific source, receiver combination. More importantly, this Data class is where we store information related to observed data and its errors. This class started in the SimPEG.Survey module, but has now been moved into its own new module SimPEG.data. See the documentation for the SimPEG.data.Data for all of the details.

Previously,

# Add data to the survey
survey.dobs = dobs
survey.std = 0.05  # a 5% relative error
survey.eps = 1.0E-6  # a noise floor

Now,

# Create a data object
data = data.Data(dobs=dobs, relative_error=0.05, noise_floor=1e-6)

You might also notice that we changed the name of the terms used to construct the standard deviation. See issue #846. Previously survey.std represented an error that was relative to the absolute value of the data. The name of this term is misleading, as it is not actually the classic statistical standard deviation.

Previously the uncertainty was constructed as:

uncertainty = survey.std * np.abs(survey.dobs) + survey.eps

We now have updated the names to be clearer and more in line with what we would naturally expect, which is accessed from data.standard_deviation. The value that is returned from this is now defined as:

data.standard_deviation = (
    data.relative_error * np.abs(data.dobs) +
    data.noise_floor
)

You can also directly set the value of data.standard_deviation if you prefer to work with that quantity.

data.standard_deviation = 0.01

This Data class is now also the object that is returned from:

data = sim.make_synthetic_data(
    m, relative_error=0.05, noise_floor=0.0, f=None, add_noise=True
)

The DataMisfit class

Previously, because the Survey class handled predicting data at the receivers, and it also had knowledge of the observed data and its noise, we constructed the data misfit measure using only the survey. Now we have specifically broken this piece up into a forward Simulation object, and a Data object. This mimics the definition of the classic data misfit measure.

\[\phi_d = \frac{1}{2}||\textbf{W}_d(\mathcal{F}(\vec{m}) - \vec{d}_{obs})||_2^2\]

The Simulation class handles the forward operation, \(\mathcal{F}\), and the Data class handles the noise, \(\textbf{W}_d=diag(\frac{1}{\sigma_i})\), and the observed data, \(\vec{d}_{obs}\). See the documentation for the SimPEG.data_misfit.L2DataMisfit for all of the details.

Previously,

# Survey knows how to predict data, knows the observed data,
# and its standard deviation
dmis = DataMisfit.l2_DataMisfit(survey)

Now,

# Create a data misfit
# The data class now knows the observed data and its standard deviation.
# The simulation knows how to create data from a model.
dmis = data_misfit.L2DataMisfit(simulation=sim, data=data)

Dask

We have begun a concerted effort to incorporate dask as a means to allow SimPEG to scale to larger computers (and take advantage of parallelism). Checkout the dask docs at https://docs.dask.org/en/latest/.

This feature is experimental at the moment and can be toggled on like so,

import SimPEG.dask

which will then enable parallel operations for a few modules. It will specifically replace these functions with dask versions,

  • SimPEG.potential_fields.BasePFSimulation.linear_operator

  • SimPEG.potential_fields.magnetics.Simulation3DIntegral.getJtJdiag

  • SimPEG.potential_fields.gravity.Simulation3DIntegral.getJtJdiag

  • SimPEG.electromagnetics.static.resistivity.simulation.BaseDCSimulation.getJ

  • SimPEG.electromagnetics.static.resistivity.simulation.BaseDCSimulation.getJtJdiag

  • SimPEG.electromagnetics.static.induced_polarization.simulation.BaseDCSimulation.getJ

  • SimPEG.electromagnetics.static.induced_polarization.simulation.BaseDCSimulation.getJtJdiag

Changelog

As can be expected, there are many changes in this release, and we hope to identify most of them here (or at least point you in the right direction).

Renamed Modules

We have taken steps to rename the modules of SimPEG to a more PEP 8 friendly system. The previous locations do not exist.

  • EM → electromagnetics

  • EM.FDEM → electromagnetics.frequency_domain

  • EM.TDEM → electromagnetics.time_domain

  • EM.NSEM → electromagnetics.natural_source

  • EM.Static → electromagnetics.static

  • EM.Static.DC → electromagnetics.static.resistivity

  • EM.Static.DC.Utils → electromagnetics.static.resistivity.utils

  • EM.Static.IP → electromagnetics.static.induced_polarization

  • EM.Static.SIP → electromagnetics.static.spectral_induced_polarization

  • EM.Static.Utils → electromagnetics.static.utils

  • EM.Utils → electromagnetics.utils

  • VRM → electromagnetics.viscous_remanent_magnetization

  • FLOW → flow

  • SEIS → seismic

  • PF → potential_fields

  • PF.Gravity → potential_fields.gravity

  • PF.GravAnalytics → potential_fields.gravity.analytics

  • PF.Magnetics → potential_fields.magnetics

  • PF.MagAnalytics → potential_fields.magnetics.analytics

  • Utils → utils

  • DataMisfit → data_misfit

  • Directives → directives

  • Fields → fields

  • InvProblem → inverse_problem

  • Inversion → inversion

  • Maps → maps

  • Models → models

  • ObjectiveFunction → objective_function

  • Optimization → optimization

  • Props → props

  • Survey → survey

  • Problem → simulation

Also generally modules with the following names have changed:

  • METHOD.SurveyMETHOD → method_name.survey

  • METHOD.SrcMETHOD → method_name.sources

  • METHOD.RxMETHOD → method_name.receivers

  • METHOD.ProblemMETHOD_xD → method_name.simulation_xd

  • METHOD.FieldsMETHOD → method_name.fields

where METHOD was the old module name for the method and method_name is the new name.

Also many of the utilities modules within these are also being deprecated. Their old names are still around and should throw a deprecation warning when loaded pointing to the correct name.

For example,

  • Utils.codeutils → utils.code_utils

  • Utils.ModelBuilder → utils.model_builder

  • EM.Utils.EMUtils → electromagnetics.utils.waveform_utils (this one is a little less obvious)

Problem to Simulation

Many of the previous Problem classes are still within the modules, but will now throw a deprecation warning which points to the updated Simulation. The renaming scheme for the Simulation-s was decided in #857.

For example,

  • Problem3D_CC → Simulation3DCellCentered

  • Problem2D_CC → Simulation2DCellCentered

  • Problem3D_e → Simulation3DElectricField

  • Problem3D_b → Simulation3DMagneticFluxDensity

  • Problem3D_h → Simulation3DMagneticField

  • Problem3D_j → Simulation3DCurrentDensity

  • etc.

Our current plan is to remove these deprecated versions in the 0.15.0 SimPEG release.

Potential field reorganization

The potential_fields module has likely received the largest amount of reorganization to make it more in line with the structure of the other modules, and some things have moved around and been renamed. There are now two separate modules within potential_fields: gravity and magnetics. All of the classes in PF.BaseGrav have been moved to potential_fields.gravity, and the classes in PF.BaseMag have been moved to potential_fields.magnetics. The Map-s that were within them have been deprecated and can instead be found in SimPEG.maps.

The option of a coordinate_system for the magnetics simulation is no longer valid and will throw an AttributeError. Instead use the SimPEG.maps.SphericalSystem.

Improvements and Additions to resistivity

We have made a few improvements to the SimPEG.electromagnetics.static.resistivity that were motivated by our work under the Geoscientists Without Borders project.

One is that we now have a 1D layered Earth simulation class, SimPEG.electromagnetics.static.resistivity.simulation_1d.Simulation1DLayers, that can be used to invert resistivity sounding data for conductivity and/or thicknesses of a set number of layers.

The second, is a new miniaturize option for the 2D and 3D resistivity simulations. This option causes the class to internally replace Dipole sources and receivers with only unique Pole sources and receivers. This can result in a dramatic speedup and reduced memory requirements when the input survey contains many more Dipole-s than electrode locations. This is especially common in the wenner type survey acquisitions that use a unique source and receiver for each measured data point. This option is disabled by default, and can be enabled by passing the minaturize=True keyword to the resistivity Simulation upon initialization.

The third is automated TreeMesh construction within the resistivity.IO class for a 2D survey.

Deprecations

Some functions and properties have been renamed to more PEP 8 friendly names, for example:

  • Survey.makeSyntheticData → Simulation.make_synthetic_data

  • Survey.srcList → Survey.source_list

  • Source.rxList → Source.receiver_list

  • Source.loc → Source.location

  • Receiver.locs → Receiver.locations

  • etc…

As mentioned before, the old names of these items will still be around, but will throw DeprecationWarnings pointing the user to the new names.

We have done work to make this release as backwards compatible as possible, but can not guarantee that all code will work. At the minimum, module imports must be changed. The old style of pair-ing a survey will still work, in which case all of the older functions will still work. This is only intended for old code to continue working, and should not be relied upon for developing new code.

As noted all of the Problem-s are being deprecated and will be removed as of version 0.15.0 of SimPEG. The deprecated utility models will also be removed then.

Pull Requests

There were 25 pull requests contributing to this release.

  • #786: Simulation class refactor.

  • #792: Use scooby for Versions.

  • #802: Simulation jk.

  • #819: Simulation jk df.

  • #835: Add simulation PF tutorials to simulation PF branch

  • #843: Fix drapeTopoLoc.

  • #844: Static receiver midpoint 3D.

  • #845: Tile map.

  • #848: Fix Ward and Hohmann example.

  • #849: Update and fix tutorials.

  • #851: Simulation method names .

  • #853: CSEM TDEM 3D example.

  • #854: Simulation dc2.5 d speedup.

  • #861: Fix typo.

  • #863: light formatting to be closer to black, update filepaths to be os independent.

  • #864: Sim dask split.

  • #865: simulation fdem testing bug.

  • #866: waveform logic bug.

  • #868: This change was necessary for fields to be called correctly for dcip2…

  • #869: Simulation tutorials 2.

  • #872: Uncertainty rename in simulation.

  • #874: Release notes for simulation as part of the Documentation.

  • #876: Simulation tutorials cleanup.

  • #879: Run black on SimPEG.

  • #882: Replace html links to documentation with local links.

Closed Issues

  • #865: FDEM: derivative test seems to ignore imag-real-flag

  • #857: DOC: Broken link

  • #850: Can’t set a waveform on initialized BaseTDEMSrc object

  • #847: Renaming of methods’ Problems

  • #846: Standard deviation and it’s place in the code base

  • #842: Static drape topo not on surface.

  • #787: Plotting integrated sensitivity for DC resistivity problem.

  • #765: DOC: Source code not linked, 404

  • #751: run black on SimPEG

  • #562: What’s the problem with Problem?

  • #527: Remove MapPair, RxPair, Pair

  • #524: Remove rxType from BaseRx

  • #202: Using the DataClass

Contributors

Combining to produce over 500 commits (listed in no particular order):