YAML settings

A YAML header is used in every .qmd file as metadata to define page behavior. The survey.qmd file in each surveydown survey also relies on the YAML header, but we’ve made some significant changes to improve the experience.

YAML omission: First, you no longer need to include any YAML header at all now as surveydown now auto-loads the following:

---
format: html
echo: false
warning: false
---

This used to be the standard YAML header in survey.qmd, but now if you leave it out, these settings will be used by default.

New YAML experience: An empty YAML is fine if you want to use all default settings, so your survey.qmd file can start directly with your library calls and survey content:

```{r}
library(surveydown)
library(other_packages)
```

Survey content...

YAML settings: We have also expanded the YAML header to include all other survey settings to customize your survey behavior. Below is a full list of all possible YAML settings with their default values, including theme, survey, and system message settings. The comprehensive list of settings used in any one survey are generated in your _survey/settings.yml file of any surveydown survey project.

---
theme-settings:
  theme: default
  barposition: top
  footer-left: ''
  footer-center: ''
  footer-right: ''

survey-settings:
  show-previous: no
  use-cookies: no
  auto-scroll: no
  rate-survey: no
  all-required: no
  start-page: your_first_page
  system-language: en
  highlight-unanswered: yes
  highlight-color: gray
  capture-metadata: yes
  required: []

system-messages:
  cancel: Cancel
  confirm-exit: Confirm Exit
  sure-exit: Are you sure you want to exit the survey?
  submit-exit: Submit and Exit
  warning: Warning
  required: Please answer all required questions before proceeding.
  rating-title: Before you go...
  rating-text: 'Rate your survey experience:'
  rating-scale: from 1-poor to 5-excellent
  previous: Previous
  next: Next
  exit: Exit Survey
  close-tab: Please close this tab manually to exit the survey.
  choose-option: Choose an option...
  click: Click here
  redirect: Redirecting in
  seconds: seconds
  new-tab: Opens in a new tab
  redirect-error: "Error: This text won't trigger any redirection..."
---

You can customize any of them in survey.qmd to fit your survey needs. For example, say you want to enable cookies, you can have these YAML settings in your survey.qmd file (cookies are on by default):

---
survey-settings:
  use-cookies: yes
---

Note also the system-messages category. Here you can customize the text of all system messages used in your survey. For example, to change the “Next” button label to “Continue”, you can do this:

---
system-messages:
  next: Continue
---

This used to be handled in a separate translations.yml file, but now they’re all consolidated into the YAML header for easier access.

Previous Button Support

One of the most requested features is finally here: previous buttons! Survey respondents can now navigate backwards through your survey, and you have full control over whether to enable this feature globally or on a per-page basis.

Global setting: By default, previous buttons are not enabled, but you can enable them globally by changing the YAML header setting in the survey.qmd file:

---
survey-settings:
  show-previous: true
---

Page-Level override: You can override the global setting on specific pages using the newly introduced sd_nav() function in your survey.qmd file:

# Enable previous button on a page
sd_nav(show_previous = TRUE)

# Disable previous button on a page
sd_nav(show_previous = FALSE)

Here are examples of previous button being enabled and disabled:

Previous button disabled

Previous button enabled

Auto Page Navigation

One of our favorite new additions is auto-injecting page navigation. In previous versions of surveydown, you needed to add sd_next() on each page to insert next buttons. But now navigation buttons are auto injected at the end of each page unless you call sd_nav() or sd_close() on that page, which will override the global settings.

This means survey page definition can be as simple as:

--- page_1

Page 1 content...

--- page_2

Page 2 content...

--- page_3

Page 3 content...

No navigation code needed - surveydown handles it for you!

Manual navigation control: Use sd_nav() or sd_close() to customize navigation.

The new sd_nav() function replaces the old sd_next(), providing a unified interface for both previous and next buttons.

In most cases, calling sd_nav() with empty parameters will suffice, as it defaults to showing the navigation buttons based on your global settings.

sd_nav() has the following arguments:

• page_next: Which page to go to next? (defaults to the next page in order)
• label_previous: Override previous button label (defaults to the system message “Previous”)
• label_next: Override next button label (defaults to the system message “Next”)
• show_previous: Show previous button? (defaults to the global setting)
• show_next: Show next button? (defaults to the global setting)

Likewise, sd_close() supports both previous and next buttons with the following arguments:

• label_close: Label on the close page buttons (defaults to the system message “Close”)
• label_previous: Override previous button label (defaults to the system message “Previous”)
• show_previous: Show previous button? (defaults to the global setting)

sd_question() Enhancements

Since surveydown is built on top of shiny, we originally used the singular word option to define multiple-choice options in sd_question() (this is what shiny uses). But this can be confusing, since most of the time you want to provide multiple options (plural).

To address this, now both option and options are supported for sd_question() when using multiple-choice type questions, such as mc, mc_buttons, etc.

Auto label to value conversion: When defining option or options, the traditional way to define options is the pattern label = value, like this:

sd_question(
  id = "favorite_color",
  type = "mc",
  label = "What's your favorite color?",
  options = c(
    "Light Blue" = "light_blue",
    "Dark Green" = "dark_green",
    "Bright Red" = "bright_red"
  )
)

But in case you forget this pattern, you can now just provide a single vector of labels like this:

sd_question(
  id = "favorite_color",
  type = "mc",
  label = "What's your favorite color?",
  options = c(
    "Light Blue",
    "Dark Green",
    "Bright Red"
  )
)

In this case, the value stored in the data will be automatically converted to snake case. In this example, the above question will store the following values in the database:

What’s Next?

This v1.0.0 release marks a major milestone for surveydown, but we’re not stopping here. We have exciting features planned for future releases. Stay tuned!

As always, we welcome your feedback and contributions. If you encounter any issues or have suggestions, please open an issue for the package, create a discussion in our organization, or contribute directly via pull requests.

Happy surveying!

The Problem: Verbose Fence Syntax

Since the beginning, surveydown has used Quarto’s fence syntax (three colons :::) to define pages. While this approach works well and leverages familiar Quarto conventions, it requires explicit opening and closing tags for every page:

::: {.sd_page id=welcome}

Welcome to our survey!

:::

::: {.sd_page id=demographics}

Please tell us about yourself.

:::

::: {.sd_page id=questions}

Now for the main questions...

:::

As your surveys grow, these closing ::: tags add visual clutter and make it harder to see where one page ends and another begins. You also need to remember to close every page - forgetting a closing fence can cause parsing errors.

The Solution: Shorthand Syntax

With surveydown package version 0.14.0, we’re introducing a new shorthand syntax that eliminates the need for closing tags. Instead, you simply use three dashes --- followed by the page ID:

--- welcome

Welcome to our survey!

--- demographics

Please tell us about yourself.

--- questions

Now for the main questions...

That’s it! Each new page delimiter automatically closes the previous page, so there’s no need to track opening and closing tags. The syntax is cleaner, more concise, and easier to scan visually.

The shorthand syntax offers several advantages:

1. Cleaner code: Eliminates visual clutter from closing tags
2. Easier to scan: Page delimiters stand out clearly, making survey structure obvious
3. Less typing: Fewer characters to type means faster survey development
4. Fewer errors: No need to remember closing tags - can’t forget what doesn’t exist!
5. Better git diffs: When you add or modify pages, diffs are cleaner without closing tag changes

Example

Let’s look at another survey example to see the improvement. Here’s the same survey written both ways:

---
format: html
echo: false
---

```{r}
library(surveydown)
```

::: {.sd_page id=welcome}

# Welcome!

Thank you for participating in our study.

```{r}
sd_next()
```

:::

::: {.sd_page id=consent}

## Informed Consent

Please read the following information carefully...

```{r}
sd_question(
  type = 'mc',
  id = 'consent',
  label = "Do you consent to participate?",
  option = c(
    'Yes, I consent' = 'yes',
    'No, I decline' = 'no'
  )
)

sd_next()
```

:::

::: {.sd_page id=demographics}

## Demographics

```{r}
sd_question(
  type = 'numeric',
  id = 'age',
  label = "What is your age?"
)

sd_question(
  type = 'mc',
  id = 'education',
  label = "What is your highest level of education?",
  option = c(
    'High school' = 'hs',
    'Bachelor\'s degree' = 'ba',
    'Graduate degree' = 'grad'
  )
)

sd_next()
```

:::

::: {.sd_page id=survey_questions}

## Main Questions

```{r}
sd_question(
  type = 'mc',
  id = 'satisfaction',
  label = "How satisfied are you with our product?",
  option = c(
    'Very satisfied' = '5',
    'Satisfied' = '4',
    'Neutral' = '3',
    'Dissatisfied' = '2',
    'Very dissatisfied' = '1'
  )
)

sd_next()
```

:::

::: {.sd_page id=end}

## Thank You!

Your responses have been recorded. You may now close this window.

```{r}
sd_close()
```

:::

---
format: html
echo: false
---

```{r}
library(surveydown)
```

--- welcome

# Welcome!

Thank you for participating in our study.

```{r}
sd_next()
```

--- consent

## Informed Consent

Please read the following information carefully...

```{r}
sd_question(
  type = 'mc',
  id = 'consent',
  label = "Do you consent to participate?",
  option = c(
    'Yes, I consent' = 'yes',
    'No, I decline' = 'no'
  )
)

sd_next()
```

--- demographics

## Demographics

```{r}
sd_question(
  type = 'numeric',
  id = 'age',
  label = "What is your age?"
)

sd_question(
  type = 'mc',
  id = 'education',
  label = "What is your highest level of education?",
  option = c(
    'High school' = 'hs',
    'Bachelor\'s degree' = 'ba',
    'Graduate degree' = 'grad'
  )
)

sd_next()
```

--- survey_questions

## Main Questions

```{r}
sd_question(
  type = 'mc',
  id = 'satisfaction',
  label = "How satisfied are you with our product?",
  option = c(
    'Very satisfied' = '5',
    'Satisfied' = '4',
    'Neutral' = '3',
    'Dissatisfied' = '2',
    'Very dissatisfied' = '1'
  )
)

sd_next()
```

--- end

## Thank You!

Your responses have been recorded. You may now close this window.

```{r}
sd_close()
```

Notice how much cleaner the shorthand version is? The page boundaries are immediately visible, and there’s no visual clutter from closing tags. Your survey structure becomes easier to understand at a glance.

Backward Compatibility

We want to emphasize that both syntaxes work perfectly in v0.14.0 and beyond. Your existing surveys using fence syntax will continue to work without any changes. We’re introducing shorthand syntax as the recommended approach going forward, but you’re free to use whichever style you prefer.

Updated Documentation

All of our documentation has been updated to use the new shorthand syntax as the primary example. However, we’ve kept documentation for both approaches:

• The Page Navigation page explains both syntaxes in detail
• The Basic Components page introduces shorthand syntax
• All our templates will be updated to use shorthand syntax

Migration Guide

The good news is you do not need to change anything in your existing surveys. They will continue to work as before. However, if you want to take advantage of the cleaner shorthand syntax, migrating is easy:

1. Replace ::: {.sd_page id=PAGE_ID} with --- PAGE_ID
2. Remove all closing ::: tags
3. Make sure each page starts with --- followed by a space and the page ID

For example, convert this:

::: {.sd_page id=page1} 

Content here

:::

::: {.sd_page id=page2}

More content

:::

To this:

--- page1

Content here

--- page2

More content

That’s all there is to it!

Looking Forward

This improvement is part of our ongoing effort to make surveydown as developer-friendly as possible. We’re constantly listening to user feedback and looking for ways to reduce friction in the survey development process.

If you have suggestions for future improvements, please let us know through GitHub Discussions or by filing an issue on our GitHub repository.

Getting Started

To start using the shorthand syntax, simply update to surveydown v0.14.0 or later:

# Install from CRAN (once 0.14.0 is released)
install.packages("surveydown")

# Or install the development version from GitHub
pak::pak("surveydown-dev/surveydown")

Then start using --- followed by your page IDs in your survey.qmd files. It’s that simple!

1 Getting Started

# Development version
pak::pak("surveydown-dev/sdstudio", ask = FALSE)

# Development version
pak::pak("surveydown-dev/surveydown", ask = FALSE)

# CRAN version
install.packages("surveydown")

sdstudio::launch()

2 The “Build” Tab

2.1 Start from a Template

The “Build” tab is your workspace of survey construction. You can either start from a prepared template or continue with your own survey files. The “Directory” will show your current working directory, but you can change it to any directory where your survey files are stored.

Note

We currently have 15 templates to choose from. The default template contains a minimum basic structure for a clean start. You might also try the question_type template for a showcase of all question types, or try some advanced features with conditional_display, conditional_navigation, etc. Our surveydown-dev GitHub organization has template repositories starting with template_. Feel free to search for your desired templates, clone or download the source code and try them out!

Start from a Template in the Build Tab

2.2 Build Your Survey

After you’ve selected your template, or if you already have an existing survey, you’ll see the dual panel of survey constructions on the “Build” tab. The “Structure” section on the left shows the page-content structure of your survey. The “Code” section on the right displays the survey scripts in a code editor.

Dual Panel of the Build Tab

2.2.1 The Structure Section

The “Structure” section on the left gives you 2 hierarchies: page and content. The pages can be toggled on to show the content, or toggled off. There are two types of content: questions and texts, with labels of “Q” and “T”, respectively.

There are buttons on each page and content which self-explain their capabilities. The “Add A New Page” button at the bottom allows you to create a new page.

The question content shows question ID, question type, and question label. You can review the full question details with the edit button, or view the full question definition code on the code section. The text content shows the first 5 words of the full piece of text.

Click on the +T/+Q button to add a text/question

All elements support drag-n-drop operations. After each operation, the structure will refresh and update the code editor on the right. If you move two test elements together, they will merge in to one text element.

Dragging two texts together will merge them

2.2.2 The Code Section

Note

For now the surveydown Studio has powerful support to survey.qmd because it is meant to be the main file for survey construction. However, even though app.R is directly editable, we still recommend that you edit it using your IDE or other text editors. All our templates have the app.R file ready to run, which can be a good starting point.

The “Code” section on the right is an integrated code editor that allows you to edit the survey scripts directly, with the survey.qmd file opened by default. It also reflects any operation you’ve done on the “Structure” section. The other file, app.R, is available for direct editing as well.

3 The “Preview” Tab

The “Preview” tab is where your survey comes to life. It provides a real-time, interactive preview of your survey as you build it, allowing you to see exactly how respondents will experience your survey.

3.1 Live Rendering

Every time you make changes in the Build tab, the survey is live-rendered and revealed in the Preview tab. It takes a couple of seconds to render, so if you switch to the Preview tab too early, you’ll see message of survey being rendered. After rendering is done, the survey will be available.

The survey preview is live and interactive

3.2 Desktop and Mobile Views

You can toggle between “Desktop View” and “Mobile View” using the pair of buttons at the top right corner:

Toggling between desktop and mobile views

3.3 New Sessions

You might have noticed there is a green refresh button on the top right as well. This is for you to manually refresh the preview if you want to restart a session:

Restart a new session using the refresh button

4 The “Responses” Tab

We built sdstudio for two purposes: for designing surveys and for managing survey response data. You can use the studio for either or both purposes.

The “Responses” tab is used to view two forms of survey response data:

1. Local Mode: This mode allows you to view “preview data” while locally previewing your survey without having a database connection. The preview data are stored in a local “preview_data.csv” file and are purely available for the purposes of previewing what your data will look like once you have a database connection.
2. DB Mode: After creating and testing your survey, you can add a database connection to start collecting survey responses in your live database.

Lets go through each of these use cases in details.

4.1 Local Mode

All our templates have Local Mode enabled by default, because we recommend you to start from local testing. To start with your local testing, you can simply go through all 3 tabs in order: Build, Preview, and then Responses.

Below is a brief walkthrough of the process:

Build, Preview, and Responses in Sequence

4.2 DB Mode

Note

For instructions on how to obtain your database credentials, see the Storing Data documentation page.

To get started with viewing your database response data, you need to firstly switch to DB Mode by clicking the “DB Mode” button on the top right corner. This is available in all 3 tabs. Then, in your “Responses” tab, you’ll see a “Database Connection” section on the top. Toggle on the “Settings” button and fill in your database credentials. You can test the connection by clicking the “Test Connection” button. If you have already set up your database credentials using sd_db_config() to create a local .env, then your credentials should show here.

In order to have a clean start, we recommend you to name a new table in the “Table” field of “Database Connection”. Upon clicking “Test Connection”, this table will only be a placeholder, and the database will create such table once you start your first survey run.

Input your credentials and create a new table

If the connection is successful, you will see a “Connected” status message in green. The “Choose a table to view” section will show the new table you just created, but all fields will be empty. They will be filled in once you run your survey in the Preview tab.

After you’ve taken care of the database connection, you can go back to the Build tab and proceed with your survey construction, following the exact same steps. The only difference is that now, when you run the survey in the Preview tab, your responses will be stored in the database table you just created instead of the local “preview_data.csv” file.

4.3 Viewing other tables

It is important to note that you can use the “Responses” tab to view all tables stored in your database. For example, if you have multiple surveys you are working on (each in a separate folder) that share the same database, you can select other tables to view from the “Responses” tab.

Input your database credentials

If the connection is successful, you will see a “Connected” status message in green. You can choose your desired response table from the dropdown menu, review its progress and details, and download the csv file if needed:

Fill in your database credentials and click on the “Test Connection” button

5 Data Security

# Database connection settings for surveydown
SD_HOST="your-database-host"
SD_PORT="your-database-port"
SD_DBNAME="your-database-name"
SD_USER="your-database-username"
SD_PASSWORD="your-database-password"
SD_TABLE="your-database-table"

# Auto mode (recommended) - tries GSSAPI first, falls back if needed
sdstudio::launch()

# Prefer mode - only use GSSAPI
sdstudio::launch(gssencmode = "prefer")

# Disable mode - forces plain connection (useful for VPN environments)
sdstudio::launch(gssencmode = "disable")

6 Conclusion

As a companion GUI for the surveydown platform, sdstudio significantly simplifies the survey creation process. It allows users to build, preview, and manage surveys through an intuitive interface, without requiring extensive programming skills. The three-tab workflow - Build, Preview, and Responses - provides a comprehensive solution that covers the entire survey lifecycle.

If you have any questions or feedback you’d like to share, please drop us a line using GitHub Discussions.

For more information, visit the sdstudio documentation or explore the surveydown platform.

1 Introduction

While surveydown works with various IDEs, the gadget features perform best in RStudio. These gadgets provide a user-friendly interface for creating survey elements without having to remember the exact syntax or function parameters.

Two primary gadgets are offered by surveydown:

1. Survey Page Gadget - Creates a new survey page
2. Survey Question Gadget - Creates a new survey question

Here is a showcase of the Survey Page Gadget in RStudio:

The Survey Question Gadget:

As you can see, these gadgets simplify the process of adding survey components. This blog walks you through how to access and use these gadgets.

2 Accessing the Gadgets

The gadgets are powered by the sd_page_gadget() and sd_question_gadget() functions, but you don’t need to call these functions directly.

You can access these gadgets in RStudio in two ways:

2.1 Using the Addins Menu

1. Click on the “Addins” dropdown menu in the RStudio toolbar
2. Type “surveydown” in the search box
3. Select either “Add Survey Page” or “Add Survey Question”

Below is a screenshot of the Addins menu with surveydown gadgets:

2.2 Keyboard Shortcuts (Recommended)

For more efficient workflow, set up keyboard shortcuts:

1. Go to Tools → Addins → Browse Addins…

2. In the Addins popup window, click on the “Keyboard shortcuts…” button on the bottom left corner.

3. Input “survey” in the search box.

4. Assign the following shortcuts:
   • Ctrl+Shift+P for the Survey Page Gadget
   • Ctrl+Shift+Q for the Survey Question Gadget
   Or if you have other preferences, feel free to customize them.

3 Using the Gadgets

3.1 Survey Page Gadget

The page gadget is straightforward:

1. Press Ctrl+Shift+P (or use the Addins menu)
2. Enter a Page ID (no spaces allowed)
3. Click “Create Page” or press Enter

This will insert a properly formatted page block at your cursor position, including an R code chunk ready for adding questions.

3.2 Survey Question Gadget

The question gadget offers more options:

1. Press Ctrl+Shift+Q (or use the Addins menu)
2. Fill in the following:
   • Question Type: Select from the dropdown (default is “Multiple Choice”)
   • Question ID: Enter a unique identifier (no spaces)
   • Question Label: Enter the actual question text
3. R Chunk Option: Check this box if you need the question to be inserted within an R code chunk
4. Click “Create Question” or press Enter

Note that if you created the question inside an existing page’s R chunk, you don’t need to check the “R Chunk Option”. If you’re adding a question elsewhere and need it to be in an R chunk, check that box.

4 Example Workflow

Here’s how a typical workflow might look:

1. Create a new page:
   • Press Ctrl+Shift+P
   • Key in your Page ID
   • Press Enter
2. Add a question:
   • Press Ctrl+Shift+Q
   • Select your desired question type
   • Key in your Question ID
   • Key in your Question Label
   • Press Enter
   • Modify your Question Options as needed
3. Add more questions as needed by repeating step 2.

By following these steps and using the gadgets, you’ll create well-structured surveys much faster than coding everything manually.

New functions / features

We added a whole bunch of new functions and features since v0.3.0, which was a major architectural change to the package. But the changes we made in v0.3.0 made it lot easier to support more features. There have been a lot, so I’m probably missing some, but here are some quick highlights that are worth calling out:

• In sd_question(), we added a new type = "matrix" to support matrix type questions.
• We added a new sd_is_answered() function to check if a question is answered or not. This is useful for conditional reactive questions defined in the server.
• We added a new sd_completion_code() function that generates a random completion code for your survey.
• We added a new sd_close() function to make a button that closes the survey.
• Stefan Munnes added a translation feature that allows you to customize the system messages in the app to any language you want.
• We modified how data handling is done so that each respondent’s state in the survey can be restored via cookies if they refresh the browser window or close and re-open the window. Before, respondents would be sent back to the start of the survey and a new entry in the database would be created, so this features allows respondents to pause and come back to the survey without that happening.
• We added two new helper functions, sd_add_question() and sd_add_page(), that make it easier to quickly create template questions and pages. They work like this:

Performance improvements

One major improvement we’ve made is to streamline how the survey.qmd file is updating. Before you needed to render the survey.qmd file yourself before running the app.R file, but now you actually never need to render it. All rendering is handled when you run the shiny app via the app.R file.

And this rendering is “smart” too - it renders the survey.qmd file into an html file, then it parses that file for all of the content needed for the survey and stores each piece in a _survey folder. This folder is then what the shiny app uses to load the survey. If no changes are detected in the survey.qmd file or any of the components in the _survey folder, then the app will always load content from the pre-rendered content in the _survey folder, which will load nearly instantaneously! Only if you make a change to your survey will it be re-rendered, and again once that rendering is done once it won’t run again until another change is detected. This results in a much more efficient app that loads nearly instantaneously.

We also improved the database updating to be much more efficient. It used to run a write operation to the database on every single question ineraction, but that slowed down the app significantly. Now that we have cookies enabled, we do the immediate record keeping in the cookies and only write to the database when the respondent closes the survey or turns the page. This is a more reasonable compromise that makes the app feel snappy while still ensuring that the data is saved even if the respondent accidentally closes the window.

Our first community contributions!

Over the past few months, we’ve been getting a lot of contributions from the community! We’ve been able to merge a few pull requests that add some new features and fix some bugs. Here are some of the highlights:

• Stefan Munnes added a new translation feature that allows you to customize the system messages in the app to any language you want. He also helped us update sd_output() to be able to output the chosen question values, chosen question option label(s), and the question label itself. This addresses feature request #128.
• Zain Hoda contributed a new auto_scroll parameter to the sd_server() function that allows you to turn off auto scrolling. This is useful if you have a lot of questions and want to make sure the respondent can see the entire question.

We’re excited to see what the community will build with surveydown! If you build something with surveydown, please let us know on the GitHub Discussions so we can highlight your work!

Motivation

In the original conception of surveydown, the entire survey was defined in a single Quarto shiny document that would render into a shiny app. While this was a rather parsimonious design, it also had some flaws that weren’t immediately apparent.

Perhaps the largest issue was page security (see this issue). With Quarto shiny documents, the qmd file is first rendered into a static html page that is then used to define the elements of the user interface in the resulting shiny app. This meant that for us to introduce pages, we relied on a combination of JS and CSS to hide and show the page divs based on users clicking on next buttons. While this resulted in a nice user experience, under the hood the entire app was still just one big html page with all of the content available.

This design meant that anyone could still see the source code to any component of the survey they wanted. A user could simply right-click and open “Inspect” then manually change the CSS of a page div from style="display: none;" to style="display: show;" and boom - the “page” would appear!

This was obviously a major security issue as pages with things like completion codes or redirect buttons at the end could be easily shown without going through the whole survey. The only solution was an architectural overhaul that would only show the content on one page at a time.

New architecture

The new architecture employs a two-file design composed of a survey and an app that renders to a traditional Shiny app:

• survey.qmd: A Quarto document that contains the survey content (pages, questions, etc), which renders to an HTML file.
• app.R: An R script defining a Shiny app that contains the global settings (libraries, database configuration, etc.) and server configuration options (e.g., conditional skipping / display, etc.).

These files must be named survey.qmd and app.R.

They typically look something like this:

---
format: html
echo: false
warning: false
---


::: {.cell}

```{.r .cell-code}
library(surveydown)
```
:::


::: {#welcome .sd-page}

# Welcome to our survey!


::: {.cell}

```{.r .cell-code}
sd_question(
  type  = 'mc',
  id    = 'penguins',
  label = "Which type of penguin do you like the best?",
  option = c(
    'Adélie'    = 'adelie',
    'Chinstrap' = 'chinstrap',
    'Gentoo'    = 'gentoo'
  )
)

sd_next(next_page = 'end')
```
:::


:::

::: {#end .sd-page}

This it the last page in the survey

:::

library(surveydown)

db <- sd_database(
  host   = "",
  dbname = "",
  port   = "",
  user   = "",
  table  = "",
  ignore = TRUE
)

server <- function(input, output, session) {

  # Define conditional skip logic here (skip to page if a condition is true)
  sd_skip_if()

  # Define  conditional display logic here (show a question if a condition is true)
  sd_show_if()

  # Main server to control the app
  sd_server()
}

shiny::shinyApp(ui = sd_ui(), server = server)

This approach allows us to separate the survey content (in the survey.qmd file) from the survey logic (in the app.R file), which comes with a few benefits:

1. Security: Since the rendered survey content is no longer directly embedded in the app as a single html page, it makes it much harder for users to tamper with the content. Only the content on one page at a time will be rendered by the server.
2. Clarity: With two files, it is now clearer where the survey content versus control logic should be defined. Before, all of the server logic was in a single server code chunk at the end of the survey.qmd file, which required the user to scroll up and down to edit the server logic versus the survey content. Now a user can have both files open in two tabs in an IDE and more easily edit the survey content and server content.
3. Simplicity: The new design eliminates the need for a Quarto extension to render the survey. This allows us to ship all of the core functionality of surveydown as a single R package, which is installed globally on your system.

The updated documentation of the Basic Components page reflects this new design.

New page architecture

The motivation to secure the page content led to a totally new approach to designing the survey pages. Our new approach actually renders the survey.qmd into a static html page and then parses it into a list of page objects. Each page object is itself a list of elements, including the page ID, question IDs, etc., as well as the rendered html content for that page.

The sd_server() function then uses this list of page objects to display one page at a time via a shiny::renderUI() function into a single “main” output. This approach allowed us to control what content is being served, eliminating the ability of survey respondents to see anything other than the content on the current page.

This approach also gave us the opportunity to overhaul how pagination works in general. Previously, users had to add a sd_next(next_page = "page_id") button at the end of each page, making sure to specify the next page to go to. This was a bit annoying as most of the time you just want to go to the next page, so specifying it felt redundant. Now users can simply add sd_next() at the bottom of each page and the server will go to the next page by default. If you want to direct the respondent to a different page, then you specify the target page using sd_next(next_page = "page_id").

Improved conditional show and skip logic

Conditionally displaying questions or skipping to pages is a core logic that many surveys need. Our original approach was relatively clunky, so since we were already introducing many breaking designs, we figured we should overhaul the logic for conditional skipping and displaying.

The new approach uses just two functions: sd_skip_if() and sd_show_if(). These functions can be provided in the main server() function in the app.R file to define the conditions and targets for either conditional displaying a question or conditionally skipping to a page. The structure for each condition in these new functions is always as follows:

<condition> ~ "target_question_id"

As an example, let’s say we want to show a question called "penguins_other" if the respondent chose the "other" option in a question called "penguins". We could do this with the following code in the app.R file:

server <- function(input, output, session) {

  sd_show_if(
    input$penguins == "other" ~ "penguins_other"
  )

  sd_server(db = db)

}

You can provide multiple conditions to the sd_show_if() function, each separated by a comma. The sd_skip_if() function works the same way, but it will skip to a target page instead of showing a target question. There is also an sd_stop_if() to stop the navigation under centain conditions. See the revised Conditional Logic page for more details on the new changes.

No more sd_config() function

One more small change we made is that the sd_config() function is no longer needed. Since we moved the conditional skip and show logic into their own functions, we took the remaining arguments that used to be provided to sd_config() and added them to the sd_server() function as options. You can now simply pass these arguments to the sd_server() function in the app.R file.

Our apologies

That’s about it for the changes with v0.3.0. We want to send our deepest apologies for anyone who has already begun a study using the orginal design. The most recent version prior to v0.3.0 was v0.2.4, so this is the version you should install if you want to stick with the old design.

That said, all development will now continue on this new design, so we strongly recommend updating to the new version and converting any existing surveys to the new design. The biggest change you’ll need to make is to move your server logic out of the survey.qmd file and into the app.R file. We’ve also updated all our templates to the new design, so you can refer to these for examples on how to convert your existing surveys.

Introduction

If you’ve never used surveydown before, take a look at the Getting Started page to get a quick introduction to the package and how to use it to make a survey.

The basic concept is this:

1. Design your survey as a Quarto document using markdown and R code.
2. Convert your survey into a Shiny app that can be hosted online and sent to respondents.
3. Store your survey responses in a Supabase database.

Getting started

For this post, we recommend starting from the demo survey available here. It provides an already working survey that you can modify to the needs of your conjoint survey.

The demo repo has a lot of files in it, but the main files defining the survey itself are:

• survey.qmd: The main body of the survey.
• app.R: The app file containing the server logic implemented in the survey, including randomizing questions, connecting to a database, etc.

Content in the survey body

After the setup code chunk where we load the surveydown package, we have a series of pages (defined with ::: fences) that include markdown-formatted text and survey questions (defined with sd_question()). You can modify any of this content as you wish to suit the needs of your survey.

In this demo, we have a few other examples included, like a conditionally displayed question (the fav_fruit question will not display if you choose “No” on the first question about liking fruit) as well as a question that skips people to the end (if you choose “blue” and not “red” on the screening page). The logic controlling the conditional display and skipping is defined with the sd_skip_if() function in the app.R file.

None of this is absolutely necessary for a conjoint survey, but often times these are features that you may want to include, such as screening people out of the survey if they don’t qualify to take it, so we include it for demonstration purposes.

Defining the choice questions

The central component of every conjoint survey is the set of randomized choice questions. To implement these in surveydown, we pre-define our choice questions in a design file that we later use in the survey to select randomized sets of choice questions to display each respondent.

We use the cbcTools package to create the pre-defined design file. The code to create the choice questions for this demo survey is in the 1_make_choice_questions.R file in the demo repo. This code generates a data frame of randomized choice questions that we then save in the project directory as choice_questions.csv.

Implementing the choice questions

The choice questions are implemented at the top of the server() function in the app.R file in the demo repo. This code does the following steps:

design <- read_csv("choice_questions.csv")

# Sample a random respondentID
respondentID <- sample(design$respID, 1)

# Store the respondentID
sd_store_value(respondentID, "respID")

# Filter for the rows for the chosen respondentID
df <- design %>%
  filter(respID == respondentID) %>%
  mutate(image = paste0("images/", image))

# Function to create the labels for a choice question
# based on the values in df

make_cbc_options <- function(df) {
  alt1 <- df |> filter(altID == 1)
  alt2 <- df |> filter(altID == 2)
  alt3 <- df |> filter(altID == 3)

  options <- c("option_1", "option_2", "option_3")

  names(options) <- c(
    glue(
      "
    **Option 1**<br>
    <img src='{alt1$image}' width=100><br>
    **Type**: {alt1$type}<br>
    **Price**: $ {alt1$price} / lb<br>
    **Freshness**: {alt1$freshness}
  "
    ),
    glue(
      "
    **Option 2**<br>
    <img src='{alt2$image}' width=100><br>
    **Type**: {alt2$type}<br>
    **Price**: $ {alt2$price} / lb<br>
    **Freshness**: {alt2$freshness}
  "
    ),
    glue(
      "
    **Option 3**<br>
    <img src='{alt3$image}' width=100><br>
    **Type**: {alt3$type}<br>
    **Price**: $ {alt3$price} / lb<br>
    **Freshness**: {alt3$freshness}
  "
    )
  )
  return(options)
}

# Create the options for each choice question

cbc1_options <- make_cbc_options(df |> filter(qID == 1))
cbc2_options <- make_cbc_options(df |> filter(qID == 2))
cbc3_options <- make_cbc_options(df |> filter(qID == 3))
cbc4_options <- make_cbc_options(df |> filter(qID == 4))
cbc5_options <- make_cbc_options(df |> filter(qID == 5))
cbc6_options <- make_cbc_options(df |> filter(qID == 6))

6. Create each choice question (6 in total)

Finally, we now have everything we need to generate each choice question. Here we’re using the mc_buttons question type so that the labels we generated will be displayed on a large button, which looks good both on a computer and phone. We give the question a unique id (e.g. cbc_q1), and a label, and then set the option to the corresponding option vector we defined above.

sd_question(
  type   = 'mc_buttons',
  id     = 'cbc_q1',
  label  = "(1 of 6) If these were your only options, which would you choose?",
  option = cbc1_options
)

# ...and 5 more questions like this

Remember that since the labels in the options are being dynamically generated on each new session (each respondent), they have to be created in the server, not in the main survey body. As a result, the sd_question() function must also be created in the server code (if you put this code in the main body, only one random set of choice options will be generated, and they’ll be the same for everyone).

To display each question in the survey body, we use sd_output("id", type = "question"), changing id to each corresponding choice question we created. In the demo survey.qmd file, you’ll see that there are 6 choice questions displayed in the main survey body (each on their own page), and each of those 6 questions are defined in the server() function in the app.R file.

When rendered, a choice question will look like this, with the values matching whatever alternative was chosen in the design file:

(1 of 6) If these were your only options, which would you choose?

Option 1

Type: Honeycrisp
Price: $ 1 / lb
Freshness: Average Option 2

Type: Fuji
Price: $ 3 / lb
Freshness: Excellent Option 3

Type: Red Delicious
Price: $ 2.5 / lb
Freshness: Average

*

And that’s it! You now have 6 randomized choice questions!

Buttons versus tables

In the example above, the conjoint choice questions are displayed as “buttons” where all the information for each alternative is shown as a button. This works particularly well for mobile phone applications where the user may need to scroll vertically to see each option.

An alternative is to use a table layout where each column represents an alternative and the row names explain the attribute. This takes a little manipulation to get it right, but the key concept is to use kable() to display the transpose of the df data frame (the subset of rows for a particular respondent). We also use the wonderful kableExtra package to modify some of the table stying. We don’t explain this code in detail in this blog post, but the gist of what we’re doing here is creating a dataframe with our alternatives and displaying it as a table with kable().

The main function doing this work in the conjoint table template is the make_cbc_table() function, which looks like this:

library(dplyr)
library(kableExtra)

# Function to create the options table for a given choice question
make_cbc_table <- function(df) {
  alts <- df |>
    mutate(
      price = paste(scales::dollar(price), "/ lb"),
      image = paste0('<img src="', image, '" width=100>')
    ) |>
    # Make nicer attribute labels
    select(
      `Option:` = altID,
      ` ` = image,
      `Price:` = price,
      `Type:` = type,
      `Freshness:` = freshness
    )
  row.names(alts) <- NULL # Drop row names

  table <- kbl(t(alts), escape = FALSE) |>
    kable_styling(
      bootstrap_options = c("striped", "hover", "condensed"),
      full_width = FALSE,
      position = "center"
    )
  function() {
    table
  }
}

It is important to note that the table just shows the options, but it doesn’t allow respondents to indicate their choice. So we still need to create a basic choice question like this:

cbc1_options <- c("option_1", "option_2", "option_3")
names(cbc1_options) <- c("Option 1", "Option 2", "Option 3")

sd_question(
  type   = 'mc_buttons',
  id     = 'cbc_q1',
  label  = "(1 of 6) If the above options were your only options, which would you choose?",
  option = cbc1_options
)

The final result is a set of choice questions that each look something like this:

Preview and check

The rest of the server() function in the app.R file has the remaining components we need, like any conditional display or skip logic. This is all standard features of any surveydown survey, so we won’t cover them in detail here and instead direct you to the documentation for details.

But before you go live, it’s a good idea to do some quick testing. You can test your survey even without having it connected to a database by setting ignore = TRUE in the sd_db_connect() function. Of course, you probably should also test it after connecting it to a database to ensure that responses are being properly stored.

When testing, you might get an error - don’t panic! Read the terminal output carefully and debug. There’s a good chance you may have missed a bug somewhere in your server code. Look in your app.R file to see if you can spot the error.

Getting the data

Once your survey is live and you start collecting responses, you can easily access your data with the sd_get_data() function. This is typically done in a separate R file (e.g., see here), which might look something like this:

library(surveydown)

# Connect to database
db <- sd_db_connect()

# Pull in the data
df <- sd_get_data(db)

For this to work, you will first have to make sure you’ve set up your database connection using sd_db_config (see here for more).

And that’s it! We hope this post was helpful, and do go check out the conjoint buttons and conjoint table templates to try out the demos yourself!

A new way to design surveys

surveydown is a flexible platform for making surveys in using three open source technologies: Quarto, shiny, and supabase. The package is still in development, but you can already use it to create surveys.

The basic concept is this:

1. Design your survey as a Quarto shiny document using markdown and R code.
2. Render your doc into a shiny app that can be hosted online and sent to respondents.
3. Store your survey responses in a supabase database.

In surveydown, your entire survey is designed using markdown and R code in a single Quarto document. There is no GUI or drag-and-drop interface - simply write plain text (markdown and R code) and boom - you have a survey!

The {surveydown} package provides a set of functions for defining the survey content and control logic. Each function starts with sd_ to make them easy to identify. You can add content to your survey using markdown formatting, or in RStudio you can edit with the visual editor. Survey questions are defined in R code chunks with the sd_question() function. Pages are defined using fences (:::), and navigation buttons handled with the sd_next() function. You can modify the control logic in the server code chunk (the last code chunk at the bottom of the .qmd file) with the sd_config() function, and you can configure the database with the sd_database().

The documentation has many more details on all of this, and later in this post I’ll give a quick overview of a few of these features. But first, let me tell you a little about why we decided to build surveydown.

Why did we build surveydown?

Do we really need another survey platform?

Like many researchers who do a lot of survey work, I’ve been frustrated with most survey platforms available. Commerical platforms like Qualtrics and SurveyMonkey are great, but they are expensive and are difficult to version control, and collaboration with others is near impossible, especially if your collaborator doesn’t have a license of their own. They also don’t allow me to own my own survey data, meaning I always am at the mercy of the platform owner. And the features are often limited. It’s not easy to do more complex things like randomization, conditional display, etc.

The only other open-source survey platform I have used is formr, which is a very powerful platform, but it is rather clunky to use (you define your survey in Google sheet cells…there’s a learning curve), and it is not as easy to edit as a simple markdown file.

What we’ve come up with is a survey platform that is flexible, relatively easy to use, and built entirely with open-source technologies. I think it solves a lot of problems, and hopefully someone out there will find it useful.

Introduction to surveydown

Getting started

After getting everything installed, we recommend starting with a template survey project. To do so, run the following in the R console:

surveydown::sd_create_survey("path/to/folder")

This will create a folder with the following files:

• example.qmd: a template survey you should edit.
• example.Rproj: An RStudio project file (helpful if you’re working in RStudio)
• _extensions: A folder with the surveydown Quarto extension needed to make everything work (don’t modify this).

If you have the example open in RStudio, you can click the “Run document” button, or in your terminal run quarto serve example.qmd. Either approach should render the example survey into a shiny app that you can preview in a browser. Don’t worry just yet about setting up your database or making the survey live - for now, we’re going to focus on designing the survey and running it locally to preview it. The example survey should look like this:

::: {#welcome .sd-page}

Page 1 content here

:::

::: {#page2 .sd-page}

Page 2 content here

:::

```{r}
sd_next(next_page = 'page2')
```

```{r}
sd_next(next_page = 'page2', label = 'Next page')
```

```{r}
sd_question(
  type  = 'mc',
  id    = 'penguins',
  label = "Which is your favorite type of penguin?",
  option = c(
    'Adélie'    = 'adelie',
    'Chinstrap' = 'chinstrap',
    'Gentoo'    = 'gentoo'
  )
)
```

```{r}
#| context: server

# Define the database settings
db <- sd_database()

# Define the configuration settings
config <- sd_config()

# The sd_server() function initiates your survey - don't change this
sd_server(
  input   = input,
  session = session,
  config  = config,
  db      = db
)
```

Features

Since surveydown is built on top of Shiny, it provides tremendous flexibility in terms of what you can do with your survey. Below are a few examples of some commons things you may want to do with your survey.

Conditional display

Let’s say we had a fourth option for “other” in our multiple choice question about penguins. If the respondent chose it, you may want a second question to popup that allows them to specify the other penguin type. To implement this, you would need to define both questions, e.g.:

```{r}
sd_question(
  type  = 'mc',
  id    = 'penguins',
  label = "Which is your favorite type of penguin?",
  option = c(
    'Adélie'    = 'adelie',
    'Chinstrap' = 'chinstrap',
    'Gentoo'    = 'gentoo',
    'Other'     = 'other'
  )
)

sd_question(
  type  = "text",
  id    = "penguins_other",
  label = "Please specify the other penguin type:"
)
```

Then in the server code chunk, you could use the show_if argument to define that the penguins_other question would only be shown if the respondent chose the other option in the penguins question, like this:

config <- sd_config(
  show_if = tibble::tribble(
    ~question_id,  ~question_value, ~target,
    "penguins",    "other",         "penguins_other"
  )
)

This will make the penguins_other question only appear if the respondent chose the other option in the penguins question, like this:

Here we’re using the tibble::tribble() function to define a data frame with three columns:

• question_id: The id of the triggering question.
• question_value: The triggering value.
• target: The id of the target question to display.

You don’t have to use tibble::tribble(), and in fact if you have a lot of show_if conditions, then you could create a csv file with all of your conditions in it and read it in to set the show_if conditions (just make sure the header has the same three column names), e.g.:

config <- sd_config(
  show_if = readr::read_csv('path/to/show_if_conditions.csv')
  )
)

```{r}
sd_question(
  type  = 'mc',
  id    = 'vehicle_ownership',
  label = "Do you own your vehicle?",
  option = c(
    'Yes' = 'yes',
    'No'  = 'no'
  )
)
```

config <- sd_config(
  skip_if = tibble::tribble(
    ~question_id,        ~question_value, ~target,
    "vehicle_ownership", "no",            "screenout"
  )
)

::: {#screenout .sd-page}

Sorry, but you are not qualified to take our survey.

:::

config <- sd_config(
  required_questions = c("vehicle_ownership", "penguins_other")
)

config <- sd_config(
  all_questions_required = TRUE
)

sd_question(
  type  = "text",
  id    = "name",
  label = "What is your name?"
)

Welcome, `r sd_display_value("name")`!

brands <- c("Toyota", "Ford", "Chevrolet", "Honda", "Nissan", "Tesla")
design <- data.frame(
  respondent_id = rep(1:10, each = 3),
  brand = unlist(lapply(1:10, function(x) sample(brands, 3, replace = FALSE)))
)
write_csv(design, "design.csv")

#>   respondent_id     brand
#> 1             1      Ford
#> 2             1     Honda
#> 3             1 Chevrolet
#> 4             2    Toyota
#> 5             2     Honda
#> 6             2    Nissan

design <- read_csv("design.csv")
resp_id <- sample(design$respondent_id, 1)
df_resp <- design %>% filter(respondent_id == resp_id)

options <- c(1, 2, 3)
names(options) <- df_resp$brand

sd_question_reactive(
  type  = "mc",
  id    = "brands",
  label = "Which of these brands do you like best?",
  option = options
)

sd_display_question(id = "brands")

sd_store_value(resp_id)

How we built it

Before I wrap up, I just want to say that I am absolutely amazed at the time we live in. I have had this idea in mind for many years, but I’m not a web developer, and I never could come up with a way to make it happen. That was actually what motivated me to write my previous blog post - it was a call for help from others!

But two things happened relatively recently that made it possible:

1. The rise of Quarto
2. The rise of LLMs

After I switched my website over from distill to Quarto, I began to learn more and more about how powerful Quarto really is for building things on the web. Then I saw the Quarto shiny document framework and I immediately thought that this just might be the missing piece I needed to make surveydown a reality. It does all the legwork of converting markdown and R code into a shiny app.

Of course, implementing this idea was still really, really hard. There were many different ways to start, and I got some excellent feedback from people in the R / Quarto dev community. Garrick Aden-Buie in particular was the first to propose the idea of using fences to denote page breaks, which was a big breakthrough early on.

But the biggest breakthrough came when I started using GPT-4 to help me brainstorm many different ideas while developing the overall platform architecture. This conversation in particular was game changing. In it, I came to solutions for multiple complex problems, including the page navigation logic and which platform to use for the database (we originally started with using Googlesheets, but ultimately decided on Supabase because it is open-source and just far easier to use).

Of course, the AI didn’t do everything. Two of my students, Pingfan Hu and Bogdan Bunea have been instrumental in helping implement many of the features the package now has, and they too have leveraged LLMs to accelerate their problem solving. Thank you guys for all of your hard work! 🙏

It’s been amazing watching this project come together over such a short period of time. The original conversations I had with GPT-4 and others in the R / Quarto community were just in March and April of this year (2024). We really didn’t start developing in earnest until the summer, and really only late June / early July at that. In just a few months, we’ve gone from an idea to a fully functional survey platform.

If you give surveydown a try, please let us know what you think! And if you find a bug or something you wished existed, please post an issue on github.

I’m so excited to see what you all will build with surveydown!