## Introduction

In this tutorial, we will build a simple 3-state Markov cohort model in hēRo3. We will compare two strategies with this model: a new treatment for a particular disease ("new"), and a scenario depicting its natural history without treatment ("nat"). In this model, patients are assumed to be in one of three mutually exclusive health states: "well", "sick", or "dead". All patients begin the model in the "well" state, and in every cycle of the model, they can either remain in the "well" state, transit to the "sick" state, or die (ie, move to the “dead” state). Patients who enter the "sick" state in a given cycle of the model may remain in that state in subsequent cycles or die; they are assumed never to return to the “well” state. We can depict this model as follows:

## Creating a New Model

To begin building this new Markov state-transition model in hēRo3, click the "Markov Cohort" button in the "New Model" section at the top of the hēRo3 Workspace, as shown below. You will be taken to the Settings page in the hēRo3 Model Editor.

## Settings

The Settings page in the hēRo3 Model Editor is the first stop on the hēRo3 Subway, and it’s where you begin the process of building a new model. Here, you can specify the country of interest (which determines how hēRo3 displays costs), the discount rates that should be used for outcomes and costs (they can differ if you like), the desired length of cycles in your model, the total number of cycles over which you would like your model to run, and the half-cycle correction method that you want to use. For this tutorial, we will specify the US as the setting of interest, and we will select annual discount rates of 3% for both outcomes and costs. We will use a cycle length of one year, and run the model for 20 cycles (ie, 20 years). Finally, we will select the "Life-Table" method of half-cycle correction, which will use the average of state membership at the start and end of each cycle as the estimate of state membership in that cycle.

## Groups

The second stop on the hēRo3 Subway in the hēRo3 Model Editor is the Groups page. The Groups page is optional, and should be completed only if you would like to incorporate heterogeneity into your model population. Since our simple model makes no assumptions with respect to population heterogeneity, we will leave the Groups page blank.

## States

The third stop on the hēRo3 Subway in the hēRo3 Model Editor is the States page, where you define the states of your model. For this particular model, we will create three health states: "well", "sick", and "dead". To create a new state to include in your model, click on the button. To ensure that patients begin the model in the "well" state, enter "1" as the initial probability for the well state, and enter "0" for every other state.

## Strategies

The fourth stop on the hēRo3 Subway in the hēRo3 Model Editor is the Strategies page, where you define the strategies you want to consider in your model. In this particular model, we will create two strategies: "nat" and "new". To create a new strategy, click on the button. The check marks next to each strategy allows you to specify whether you want to include a particular strategy in a given model run. At least two strategies must be checked for a hēRo3 model to run. You may enter as many strategies as you like on this page.

## Parameters

Before moving to the fifth stop on the hēRo3 Subway in the hēRo3 Model Editor, we will first take a detour to visit the hēRo3 Parameters page. The Parameters page is where you define all of the variables that you want to use in your model. Variables can be either inputs or intermediate calculations. You specify the value that every variable takes on as an R expression in the Formula column on this page. A variable's value can be either a constant or a formula. Once you define a variable, you can reference it elsewhere in your model. Variables are easy to spot in hēRo3, as they are always represented as a blue pill. To create a new variable, click on the 'Add' button. In most models, you need to define variables for transition probabilities, health-state utilities, and costs, as shown in the image below.

## Transitions

Now that we have defined the variables that we want to use in our model, we can return to the fifth stop on the hēRo3 Subway in the hēRo3 Model Editor, which is the Transitions page. When you are building a Markov cohort model, the Transitions page is where you define the transition matrices for each strategy in your model. In hēRo3, you define transition matrices using a so-called “long-form” table, where each row represents a cell in the transition matrix for one or more strategies, and the value in the formula column is an R expression representing the transition probability. You only need to define transition probabilities that are possible (eg, you do not need to specify the probability of transition from Dead → Well); if you do not enter a transition probability between two states, hēRo3 assumes that its value is zero. For this particular model, this means that we only need to define the following transitions:

- Well → Well
- Well → Sick
- Well → Dead
- Sick → Sick
- Sick → Dead
- Dead → Dead

In the Formula column, you reference transition probabilities that you created on the Parameters Page. Once you have described the probabilities of transitioning from a particular state to all other states, and you want to specify the probability of staying in the starting state, you can simply use the keyword, 'C', which stands for “complement” in hēRo3. When you specify a transition probability as “C”, hēRo3 automatically calculates its value as one minus all other probabilities of transition from that state, ensuring that the sum of all your probabilities always equals one.

## Health and Economic Values

The sixth stop on the hēRo3 Subway in the hēRo3 Model Editor is the Values page The Values page is divided into two sections: Health Values and Economic Values.

### Health Values

Health Values is where you define the line-item outcomes that you want to account for in your model. For the model that we are building in this example, we want to include life-years and quality-adjusted life-years (QALYs) by health state. We need to designate a name for each health value, provide a description for it, and describe the different ways in which it can accrue from an accounting standpoint. Each entry must be associated with a given health state (or all health states), a given strategy (or all strategies), and a formula representing the amount of this particular outcome that accrues in each cycle. Consider, for example, the health value, "well_qalys", which represents QALYs accrued in the "well" state. Since "nat" and "new" have different utility values for the “well” state, we need to specify two entries (ie, one for each strategy), with each referencing the corresponding's strategy's utility value in the "well" state. To add a new health value, click the button. To add new entry within an existing health value, click the button next to that value.

### Economic Values

The accounting framework in Economic Values works in exactly the same manner as it does in Health Values, but it represents line-item costs that you want to account for in your model. For this particular model, we want to include costs of medication and terminal care. While values most often are associated with being resident in a particular state, sometimes you may want to assign values to a particular transition between states. In this particular model, we want to define one-time costs of terminal care as being associated with transitions from the “sick” state to the “dead” state. You can do this in hēRo3 by selecting "sick → dead" under the state column, rather than a single health state.

## Summaries

The seventh and final stop on the hēRo3 Subway in the hēRo3 Model Editor is the Summaries page. Summaries allow you to define the aggregate measures of outcomes and costs that you want to include in your model. As with Values, summaries is divided into two sections: Health Summaries and Economic Summaries.

### Health Summaries

In Health Summaries, you need to specify the aggregate outcome measures that you want to include in your final tallies of model outcomes. Each health summary is a health outcome that you can display as a separate result of your model; you also can use any health summary as a denominator in a cost-effectiveness ratio. In this particular model, we want to define health summaries for life-years and QALYs. For each such measure, we need to designate a name, an initial value for willingness-to-pay for this outcome, and the set of health values that together comprise it. For example, in this particular model, we have given total life-years the name "lys", we have assigned it an initial willingness-to-pay value of $100,000 per life-year saved, and we have specified that it is equal to the sum of "well_lys" and "sick_lys" (the health values that we earlier defined that tallied life-years in the “well” and “sick” states). To add a new health summary, click the button. To add new entry within an existing health summary, click the button next to that summary.

### Economic Summaries

Economic summaries work similarly to health summaries, but represent components of total cost that we want to use in our model. Each economic summary is an aggregate measure of cost that can be displayed as a separate result of your model; you also can use any economic summary as the numerator in a cost-effectiveness ratio. For our model, we will define a single economic summary, which is the sum of "med_cost" and "term_cost".

## Results & Sensitivity Analyses

Now that we have completed all of the steps required to build a Markov cohort model in hēRo3, we can go to the base case results page and see the output from our model. We can also set up deterministic sensitivity analyses and probabilistic sensitivity analyses, and solve for a value-based price. When a model fails to run, a red error message will be displayed at the bottom of the screen. A list of common error messages and what they mean can be found here.