Design

This page documents the general design of fastreg. It covers some requirements, the public-facing interface, and some diagrams highlighting the general flow of the main functions.

Requirements

The core requirements of fastreg are to:

1. Convert Danish register data from SAS files to the modern and efficient Parquet format.
2. Read register Parquet files into R as a DuckDB table.
3. Provide a targets pipeline template to convert multiple registers in parallel.
4. Provide functions to list available SAS or Parquet register files directly from R.

Interface

The interface (the functions and objects that are exposed to users) is based on some specific naming conventions. Specifically, we generally name function by the action they perform and the object(s) they perform it on in the format {action}_{object}(). Actions are verbs that describe what a function does, while objects are nouns that represent the objects that the functions operate on. Below is an overview of the main actions and objects within fastreg.

The actions are:

• convert: Convert a register SAS file (or multiple) to Parquet.
• list: List files in a directory, e.g., SAS or Parquet files.
• read: Read a Parquet register into R as a DuckDB table.
• use: Use a template in the current project.

While the objects are:

• chunk_size: Number of rows to read per chunk during conversion.
• path: A character vector of one or more paths.
• output_dir: The directory to save the Parquet output to.

Converting SAS files from a single register

flowchart TD
    identify_paths("Identify register path(s)<br>with list_sas_files(path)")
    path[/"path<br>[Character vector]"/]
    output_dir[/"output_dir<br>[Character scalar]"/]
    chunk_size[/"chunk_size<br>[Integer scalar]"/]
    convert_register("convert_register()")
    output[/"Parquet file(s)<br>written to output_dir"/]

    %% Edges
    identify_paths -.-> path --> convert_register
    output_dir & chunk_size --> convert_register
    convert_register --> output

    %% Style
    style identify_paths fill:#FFFFFF, color:#000000, stroke-dasharray: 5 5

Converting multiple registers in parallel

flowchart TD
    copy_pipeline("use_targets_template()")
    edit["Edit _targets.R as needed"]
    run_pipeline("targets::tar_make()")
    output[/"Parquet file(s)<br>written to directory<br>specified in _targets.R"/]

    %% Edges
    copy_pipeline --> edit --> run_pipeline --> output

    %% Style
    style edit fill:#FFFFFF, color:#000000, stroke-dasharray: 5 5

Reading a Parquet register

flowchart TD
    path[/"path<br>[Character scalar]"/]
    read_register("read_register()")
    output[/"Output<br>[DuckDB table]"/]

    %% Edges
    path --> read_register --> output