New architecture in v0.3.0 (and loads of breaking changes)!

We’re releasing v0.3.0, and with it multiple breaking changes.

The surveydown package is only a couple months old, and thanks to many early users we learned about some design flaws that needed to be fixed. As a result, with the release of {surveydown} 0.3.0, the package has undergone a major overhaul to improve security, modularity, and extensibility.

We introduced several major breaking changes in this release, so we felt we should make a blog post to both explain why we felt these changes were needed as well as introduce the new architecture we have adopted.

Motivation

In the original conception of surveydown, the entire survey was defined in a single Quarto shiny document that would render into a shiny app. While this was a rather parsimonious design, it also had some flaws that weren’t immediately apparent.

Perhaps the largest issue was page security (see this issue). With Quarto shiny documents, the qmd file is first rendered into a static html page that is then used to define the elements of the user interface in the resulting shiny app. This meant that for us to introduce pages, we relied on a combination of JS and CSS to hide and show the page divs based on users clicking on next buttons. While this resulted in a nice user experience, under the hood the entire app was still just one big html page with all of the content available.

This design meant that anyone could still see the source code to any component of the survey they wanted. A user could simply right-click and open “Inspect” then manually change the CSS of a page div from style="display: none;" to style="display: show;" and boom - the “page” would appear!

This was obviously a major security issue as pages with things like completion codes or redirect buttons at the end could be easily shown without going through the whole survey. The only solution was an architectural overhaul that would only show the content on one page at a time.

New architecture

The new architecture employs a two-file design composed of a survey and an app that renders to a traditional Shiny app:

• survey.qmd: A Quarto document that contains the survey content (pages, questions, etc), which renders to an HTML file.
• app.R: An R script defining a Shiny app that contains the global settings (libraries, database configuration, etc.) and server configuration options (e.g., conditional skipping / display, etc.).

These files must be named survey.qmd and app.R.

They typically look something like this:

survey.qmd file

---
format: html
echo: false
warning: false
---

```{r}
library(surveydown)
```

::: {#welcome .sd-page}

# Welcome to our survey!

```{r}
sd_question(
  type  = 'mc',
  id    = 'penguins',
  label = "Which type of penguin do you like the best?",
  option = c(
    'Adélie'    = 'adelie',
    'Chinstrap' = 'chinstrap',
    'Gentoo'    = 'gentoo'
  )
)

sd_next(next_page = 'end')
```

:::

::: {#end .sd-page}

This it the last page in the survey

:::

app.R file

library(surveydown)

db <- sd_database(
  host   = "",
  dbname = "",
  port   = "",
  user   = "",
  table  = "",
  ignore = TRUE
)

server <- function(input, output, session) {

  # Define conditional skip logic here (skip to page if a condition is true)
  sd_skip_if()

  # Define  conditional display logic here (show a question if a condition is true)
  sd_show_if()

  # Main server to control the app
  sd_server()
}

shiny::shinyApp(ui = sd_ui(), server = server)

This approach allows us to separate the survey content (in the survey.qmd file) from the survey logic (in the app.R file), which comes with a few benefits:

1. Security: Since the rendered survey content is no longer directly embedded in the app as a single html page, it makes it much harder for users to tamper with the content. Only the content on one page at a time will be rendered by the server.
2. Clarity: With two files, it is now clearer where the survey content versus control logic should be defined. Before, all of the server logic was in a single server code chunk at the end of the survey.qmd file, which required the user to scroll up and down to edit the server logic versus the survey content. Now a user can have both files open in two tabs in an IDE and more easily edit the survey content and server content.
3. Simplicity: The new design eliminates the need for a Quarto extension to render the survey. This allows us to ship all of the core functionality of surveydown as a single R package, which is installed globally on your system.

The updated documentation of the Survey Components page reflects this new design.

New page architecture

The motivation to secure the page content led to a totally new approach to designing the survey pages. Our new approach actually renders the survey.qmd into a static html page and then parses it into a list of page objects. Each page object is itself a list of elements, including the page ID, question IDs, etc., as well as the rendered html content for that page.

The sd_server() function then uses this list of page objects to display one page at a time via a shiny::renderUI() function into a single “main” output. This approach allowed us to control what content is being served, eliminating the ability of survey respondents to see anything other than the content on the current page.

This approach also gave us the opportunity to overhaul how pagination works in general. Previously, users had to add a sd_next(next_page = "page_id") button at the end of each page, making sure to specify the next page to go to. This was a bit annoying as most of the time you just want to go to the next page, so specifying it felt redundant. Now users can simply add sd_next() at the bottom of each page and the server will go to the next page by default. If you want to direct the respondent to a different page, then you specify the target page using sd_next(next_page = "page_id").

Improved conditional show and skip logic

Conditionally displaying questions or skipping to pages is a core logic that many surveys need. Our original approach was relatively clunky, so since we were already introducing many breaking designs, we figured we should overhaul the logic for conditional skipping and displaying.

The new approach uses just two functions: sd_skip_if() and sd_show_if(). These functions can be provided in the main server() function in the app.R file to define the conditions and targets for either conditional displaying a question or conditionally skipping to a page. The structure for each condition in these new functions is always as follows:

<condition> ~ "target_question_id"

As an example, let’s say we want to show a question called "penguins_other" if the respondent chose the "other" option in a question called "penguins". We could do this with the following code in the app.R file:

server <- function(input, output, session) {

  sd_show_if(
    input$penguins == "other" ~ "penguins_other"
  )

  sd_server(db = db)

}

You can provide multiple conditions to the sd_show_if() function, each separated by a comma. The sd_skip_if() function works the same way, but it will skip to a target page instead of showing a target question. See the revised Configuration Options page for more details on the new changes.

No more sd_config() function

One more small change we made is that the sd_config() function is no longer needed. Since we moved the conditional skip and show logic into their own functions, we took the remaining arguments that used to be provided to sd_config() and added them to the sd_server() function as options. You can now simply pass these arguments to the sd_server() function in the app.R file.

Our apologies

That’s about it for the changes with v0.3.0. We want to send our deepest apologies for anyone who has already begun a study using the orginal design. The most recent version prior to v0.3.0 was v0.2.4, so this is the version you should install if you want to stick with the old design.

That said, all development will now continue on this new design, so we strongly recommend updating to the new version and converting any existing surveys to the new design. The biggest change you’ll need to make is to move your server logic out of the survey.qmd file and into the app.R file. We’ve also updated all our demos to the new design, so you can refer to these for examples on how to convert your existing surveys.