Unlocking Collaborative Power with Git, GitHub CI/CD, and LLMs in Pharma

Eli Miller, Ben Straub, Eric Nantz

Objectives

• Learn about Tools that have been used on the R Consortium FDA Pilots to collaborate and to ensure accuracy and reproducibility.
  • renv, git/GitHub, LLMs, CI/CD
• Learn more about the Pilots and their role in helping to make R and open source a viable option for delivering safe and effective drug submissions.

Intro to R Consortium

• Submission Working Group meets the first Friday of every month. Open to all!
• Click here to learn how to join.

Intro to R Consortium R-based Submission Pilots to FDA

Problem Statement

• How do people in the Pilots collaborate, ensure accuracy and reproducibility while working across multiple companies, platforms, OS, timezone and skill sets?

• renv

• git, GitHub

• CI/CD

• LLMs

Reproducibile Environments

Reproducibile Environments with Open Source Products

Reproducibile Environments with Open Source Products

• A drug company uses open source products, like R, to demonstrate that a drug is safe and effective.

• How can the company ensure the analysis is reproducible when given to a regulatory agency?

renv: Reproducible R Package Environments

• Isolates project-specific R package libraries
• Captures and locks package versions for reproducibility
  • renv.lock
  • FDA uses this to rebuild environment and LLMs consume to build the ADRG
  • The Pilots did/are using this - just in case this wasn’t clear!! renv.lock is the anchor for everything.
• Simplifies dependency management and restoration
  • Packages depend on other packages!!
• Integrates smoothly with RStudio and other tools
• Facilitates collaboration by sharing a consistent environment

Peek at the renv.lock

{
  "R": {
    "Version": "4.4.3",
    "Repositories": [
      {
        "Name": "CRAN",
        "URL": "https://packagemanager.posit.co/cran"
      }
    ]
  },
  "Packages": {
    "MASS": {
      "Package": "MASS",
      "Version": "7.3-65",
      "Source": "Repository",
      "Priority": "recommended",
      "Date": "2025-02-19",
      "Revision": "$Rev: 3681 $",
      "Depends": [
        "R (>= 4.4.0)",
        "grDevices",
        "graphics",
        "stats",
        "utils"
      ],
      "Imports": [
        "methods"
      ],
      "Suggests": [
        "lattice",
        "nlme",
        "nnet",
        "survival"
      ],
      "Authors@R": "c(person(\"Brian\", \"Ripley\", role = c(\"aut\", \"cre\", \"cph\"), email = \"Brian.Ripley@R-project.org\"), person(\"Bill\", \"Venables\", role = c(\"aut\", \"cph\")), person(c(\"Douglas\", \"M.\"), \"Bates\", role = \"ctb\"), person(\"Kurt\", \"Hornik\", role = \"trl\", comment = \"partial port ca 1998\"), person(\"Albrecht\", \"Gebhardt\", role = \"trl\", comment = \"partial port ca 1998\"), person(\"David\", \"Firth\", role = \"ctb\", comment = \"support functions for polr\"))",
      "Description": "Functions and datasets to support Venables and Ripley, \"Modern Applied Statistics with S\" (4th edition, 2002).",
      "Title": "Support Functions and Datasets for Venables and Ripley's MASS",
      "LazyData": "yes",
      "ByteCompile": "yes",
      "License": "GPL-2 | GPL-3",
      "URL": "http://www.stats.ox.ac.uk/pub/MASS4/",
      "Contact": "<MASS@stats.ox.ac.uk>",
      "NeedsCompilation": "yes",
      "Author": "Brian Ripley [aut, cre, cph], Bill Venables [aut, cph], Douglas M. Bates [ctb], Kurt Hornik [trl] (partial port ca 1998), Albrecht Gebhardt [trl] (partial port ca 1998), David Firth [ctb] (support functions for polr)",
      "Maintainer": "Brian Ripley <Brian.Ripley@R-project.org>",
      "Repository": "CRAN"
    }
...

4 key renv commands

renv.R

renv::init()

renv::status()

renv::snapshot()

renv::remove()

• Initializes a new renv project by creating a project-specific library and a lockfile.

• Checks for any discrepancies between the state of the project’s library and the renv.lock file.

• Captures the state of your current project environment and writes it to the renv.lock file.

• Removes one or more packages from your project’s library, making it easy to manage unused dependencies.

renv tips and tricks

• We don’t need to deliver every package to regulatory agencies, e.g. as we have built custom solutions to help with our processes we realized these are not actually used in the analysis. How can we ignore them!?!

• .renvignore is a big win here! Just like a .gitignore

• Don’t edit the renv.lock file manually - use the nice functions!

• Don’t be scarred to update to latest packages either - but use the nice functions!

LLM & ADRG with btw

btw(
  pkg,
  "write an one sentence description 
  on this package's functionality."
)

Here Ning used the btw package which wraps around the ellmer package and can inject your R environment and objects so an LLM can understand them. R objects, packages, functions, and documentation can be fed in to describe what your code in R is actually doing. On the right you see…

• Thanks Ning!

LLM & ADRG with Ellmer

Manually generated ADRG from pilot 3

🤖LLM generated ADRG in pilot 5

One of the ways we utilized LLM in pilot 5.

Please review the following R code and identify the variables used, along with the corresponding datasets they belong to. …”

“Please review the following R code and identify the filtering criteria applied…”

• Courtesy of Ning Leng ✨

Git/GitHub

Gitflow

Our goals:

• To maximize useful dev work,

• to maximize code quality, and

• to maximize code velocity.

In the past years I’ve seen a lot of different ways of using Git. Found a few main benifits from a well crafted gitflow.

Max Dev work. Biggest tripping point I see. Make sure all parts of the gitflow are needed and not just in place because you see others do it. Devleopers should not need to shepard their code through the gitflow process, they should only need to focus on developing and testing.

Max Code quality. test driven development can help. Make sure tests are run, all value-add quality checks are there. and codeowners are doing meaningful reviews

Max Code Velocity. Make sure all steps are needed and everyone knows who is reviewing, reviewing for what, and why they’re reviewing. Tests are running quickly, code is deployed quickly. And releases and tags are made available

Build from the ground up with a focus on quality, development, and users. Don’t implement because you see it on this slide, or saw someone else do it

On the left I have an example of an easy gitflow. Even this one could be overengineered.

To start your development efforts are in their own seperate feature branches, which are then merged into a staging environment. I see a lot of QA work being done in staging environments before it gets pushed into production. Once approaved in stage, it can have a final round of checks done, pushed to the main branch, deployed to production, and tagged.

• Nonauthoritative, only take the parts you need!

Pull Requests and Collaboration

Code moving between branches or a transision of “state.” A structured way for reviewing code

Here is an example of a pull request in pilot 5 and you can see a few different things in the pull request process. You can see Ben commenting the team and individual contriutors, Checks successfully passing, and some automation in the GitHub project to help us prioritize work

Code Quality Automated

• 🕺Code Style
• 🧹Linting
• Spelling
• 📝Testing
• 📊Code Coverage
• ✅Build Check

Diving in deeper, here are some examples of some workflows that we run. A lot of these are taken from r-lib, pharmaverse, and other contrubutors. We check for code style, code linting(which is different somehow), a spell check, testing, test coverage, and an R CMD check. Before any code gets into the main branch it has to pass these tests. When doing development you get very clear and actionalbe feedback on what to do.

Code Quality and AI

💬 A structured chat

tlf-qc.qmd

type <- type_object(
  diff_list = type_string(),
  conclusion = type_boolean(),
  diff_score = type_integer()
)
res <- chat$chat_structured(
  prompt,
  new_image,
  comp_image,
  type = type_qc_diffs
)

🤖 and a structured result

Above we had some of the QC checks for pilot 5. One of the issues we had with some of the pilots were … And we thought to lean on some LLM to do the QC checking. Like Ning’s example above, this uses the ellmer package to give us a stucutred result that could give us a list of differneces to comment to the users, a boolean result on whether or not the TLFs are the same, and a score noting how different they are.

This is sent to an LLM which evaluates the two images, describes the differences, determines if these differences are material, then a comment is added to note all of these.

🚀Deployments

Code being deployed out to the world

Finally, once code is pushed into a staging or main branch, the code should be ready to go. Part of a good workflow should be to automatically push your approved code to a deployment environment. Doing this manually can be error prone and can work. In this example you can see some documentation of this presentation being deployed! You can see the most active version, who deployed it, and when.

Manual Work

Assemble Submission Pilot 5 files to meet eCTD specifications:

• Create folder structure
• Compile ADRG and cover letter
• Move datasets and programs to correct location
• Update file names to comply with requirements
• Push updates to separate GitHub repository

Paper with eCTD folder structure graphic https://www.lexjansen.com/phuse-us/2019/sa/SA04.pdf

Easy to make a mistake and miss a key step!

• Easy to get frustrated while trying to get these pilots actually working
• Credit: https://media3.giphy.com/media/v1.Y2lkPTc5MGI3NjExczRkYWJnN3ZmYzZsYjZpNWNlNzJ0eWNvMnR2eWRmN3RibXV5MnVzayZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw/ReImZejkBnqYU/giphy.gif

GitHub Actions to the Rescue!

• Run arbritrary commands or scripts inside workflow
• Launch on key events (updates to main branch) or schedule (cron)
• Perform Git operations on our behalf

Image credit: LEGO The Office set https://www.lego.com/en-us/product/the-office-21336

Bash Scripting

create-ectd-bundle.sh

# Define file and directory paths
ECTD_BUNDLE_DIR=submissions-pilot5-datasetjson-to-fda
ECTD_LETTER_DIR=${ECTD_BUNDLE_DIR}/m1/us
ECTD_ROOT_DIR=${ECTD_BUNDLE_DIR}/m5/datasets/rconsortiumpilot4container/analysis/adam

# additional paths ...

# Create directory structure for ectd bundle
mkdir -p "${ECTD_LETTER_DIR}"
mkdir -p "${ECTD_PROGRAMS_DIR}"
mkdir -p "${ECTD_ADAM_DATASETS_DIR}"
mkdir -p "${ECTD_SDTM_DATASETS_DIR}"

# example: copy helper fcns script
if [ -f "${PROGRAMS_SOURCE_DIR}/pilot5-helper-fcns.r" ]; then
cp "${PROGRAMS_SOURCE_DIR}/pilot5-helper-fcns.r" "${PROGRAMS_DESTINATION_DIR}/."
echo "pilot 5 helper script copied to ${PROGRAMS_DESTINATION_DIR}/pilot5-helper-fcns.r"
fi

# additional operations ...

Bash Scripting (continued)

push-ectd-bundle.sh

# Set up file paths
ECTD_REPO_DIR=submissions-pilot5-datasetjson-to-fda

# Navigate to clone of ECTD repo
cd "./${ECTD_REPO_DIR}"
echo "Open root of ECTD GitHub Repo"

# Commit updated bundle to repository
git add -A .
git config user.name github-actions
git config user.email github-actions@github.com
git commit -am "GH Action: Update Pilot 5 ECTD bundle"
git push --set-upstream origin main

echo "Updated ECTD bundle successfully to repository"

GitHub Action Workflow Steps

Run automatically on each push to main branch event

publish-ectd-bundle.yaml

- name: Create ECTD bundle
  run: bash ./.github/scripts/create-ectd-bundle.sh

- name: Archive ECTD bundle
  uses: actions/upload-artifact@v4
  with:
    name: ectd-bundle-archive
    path: submissions-pilot5-datasetjson-to-fda

- name: Push ECTD bundle repository
  run: bash ./.github/scripts/push-ectd-bundle.sh

Automation for the Win

The Wrap Up

• All volunteer group from across Pharma and CROs coming together to do R-based Submissions to the FDA.
• We are always experimenting and looking for ways to implement new ways of working and latest technology and processes.
• Get involved! More info here.