Access the full webinar here

Artificial Intelligence (AI) and Machine Learning (ML) are no longer future goals for life sciences—they’re current necessities. In this session, PTP joined John Conway, Chief Visioneer Officer of 20/15 Visioneers, to discuss the practical realities of implementing AI in research environments. Titled “The Rational AI Architect,” this webinar explores how biotech companies and research teams can lay the groundwork for successful AI projects.

The AI “Gold Rush” in Life Sciences

In 2023 and 2024, nearly every technology roadmap has been updated to include generative AI, LLMs, or predictive modeling. But for life sciences companies—especially those still modernizing infrastructure—AI can be more aspiration than reality. Cloud data migration and governance must come first, and the foundation must be secure, scalable, and compliant with life sciences standards like GxP and HIPAA.

PTP’s team has helped companies design for long-term success by aligning IT infrastructure for AI workloads, assessing readiness, and educating internal teams. This blog summarizes key takeaways from the conversation with John Conway and Aaron Jeskey, Senior Cloud Architect at PTP.

Where AI Initiatives Begin—and Why It Matters

AI and ML requests in life sciences organizations typically originate in one of two ways:

Bottom-Up: From Researchers to Leadership

Requests often come from bench scientists, data analysts, or bioinformaticians who see the potential of AI in their workflows. While technically promising, these initiatives struggle without organizational support. Challenges include:

• Lack of platform consistency

• Poorly defined processes for infrastructure management

• Limited buy-in from senior leadership

• Fragmented tools and shadow IT

To scale successfully, research IT teams need alignment from leadership on governance, security, and funding.

Top-Down: From Leadership to Technical Teams

When executives lead the charge for AI, expectations tend to be high—but often misaligned with technical feasibility. Challenges include:

• A shortage of qualified cloud data engineers or DevOps resources

• Unrealistic timelines

• Missing foundational components (e.g., data orchestration, monitoring, tagging strategies)

• Security and compliance blind spots

Without sufficient technical resources or internal education, the disconnect between strategy and execution grows quickly.

Managing Expectations and Building Real Platforms

Whether AI demand comes from the C-suite or the lab, the risks are similar: misalignment, wasted investment, and failed pilots. To mitigate this, PTP recommends:

• Starting with a Well-Architected Framework Review to benchmark infrastructure readiness

• Implementing cloud governance and cost control early

• Leveraging scalable platforms like AWS for Life Sciences to support evolving workloads

• Building internal fluency through education and transparent reporting

In one PTP case study, a genomics company attempted to implement ML without a centralized data repository. After a platform redesign and cloud engineering support, they reduced duplication, improved compliance, and accelerated model training by 30%.

Conclusion: Rational AI Starts with Realistic Infrastructure

Whether requests come top-down or bottom-up, life sciences companies need a unified strategy for implementing AI. That means prioritizing foundational architecture, building in compliance from day one, and maintaining strong communication between research and leadership.

PTP helps clients develop cloud-native AI platforms that are secure, scalable, and optimized for scientific research. From managed IT for labs to multi-region HPC clusters, we help life sciences companies move from AI talk to AI outcomes.

PTP joins with John Conway of 20/15 Visioneers to discuss the path for AI in Life Sciences.  2023 and 2024 have brought AI into almost every technology conversation.  In most instances, the AI discussion is about “how can we harness the value in the future”?  For drug discovery, most data environments will have to walk before they run.

Download the latest white paper on Scientific Data Management: Best Practices to Achieve R&D Operational Excellence

In this webinar PTP and 20/15 Visioneers will discuss:

• AI – the “it” word for 2024
• What is “The Rational AI Architect”
• Best Practices for Early Stage Life Sciences data environments
• Case studies of PTP getting data “ready”
• Roadmap for leveraging AI

Artificial Intelligence (AI) is transforming industries across the board, and life sciences is no exception. From drug discovery to gene therapy, AI is enabling significant efficiency gains. However, with AI’s potential comes a need for careful planning and execution. 

The Role of the Rational AI Architect

The concept of the rational AI architect revolves around building AI solutions with a focus on first principles. This approach aims to create robust AI solutions that are not over-engineered or overly complex. It emphasizes understanding the core problem, assessing existing resources, and developing a strategy to build efficient AI systems.

The rational AI architect avoids getting swept up in the hype surrounding AI, focusing on the foundational work required to ensure AI’s success in the life sciences field. This includes data acquisition, management, and security practices, along with considerations for the cultural shift required to adopt AI effectively.

Building a Culture of Data-Driven Decision-Making

A successful AI initiative starts with a culture that views data as an asset. Organizations need to cultivate a mindset where data is seen as a valuable resource that requires proper governance and management. Without this cultural foundation, AI projects are likely to encounter significant obstacles.

The panelists emphasized the importance of having a clear data strategy that encompasses data tagging, contextualization, and versioning. By establishing these practices, life sciences companies can build a foundation that supports AI and machine learning (ML) initiatives. As one panelist mentioned, “data is currency,” and managing it effectively is key to success.

Key Challenges and Solutions

The challenges associated with AI in life sciences can be broadly categorized into cultural, data-related, and security challenges. Here’s a breakdown of some key takeaways from the discussion:

• Cultural Challenges: Creating a culture that values data and understands the importance of AI requires effort. Organizations should establish incentives to encourage proper data management and compliance.
• Data Challenges: To achieve AI success, companies must ensure that their data is findable, accessible, interoperable, and reusable (FAIR). This includes developing a scientific data strategy, assessing the health of existing data, and maintaining proper versioning and tagging practices.
• Security Challenges: Security is a top concern in AI adoption. It’s crucial to ensure that sensitive data is protected, and the right protocols are in place to maintain compliance with industry regulations. Implementing a comprehensive security strategy is a fundamental step.

Establishing an AI Center of Excellence

To ensure successful AI adoption, the panelists recommended establishing an AI Center of Excellence. This approach brings together a team of experts to oversee AI initiatives, ensuring that they align with organizational goals and best practices. The AI Center of Excellence can help:

• Define the scope of AI projects and set clear objectives.
• Provide guidance on foundational model selection.
• Ensure compliance with data governance and security protocols.
• Facilitate cross-functional collaboration and knowledge sharing.

The Path Forward

AI has the potential to revolutionize life sciences, but it requires a thoughtful approach. Organizations should start small, focus on building a solid data foundation, and establish the right culture to support AI adoption. The rational AI architect approach encourages a stepwise, iterative process that prioritizes data quality, security, and stakeholder collaboration.

In the fascinating, high risk/high reward domain of biotechnology, research validation and landing the next wave of funding stand as crucial checkpoints in first years of a startup. The 2023 Life Sciences arena has witnessed escalating scrutiny, disturbing reports of manipulated outcomes, and pressure to deliver results faster than ever before. Because of these pressures, building efficient, transparent and reliable data pipelines, in this case with Terraform, has become a top priority for leadership teams across the industry.

In an effort to address the data authenticity priority, PTP’s life sciences client selected Amazon Web Services (AWS) as their strategic consulting partner. From the beginning, the client’s primary goal in their engagement with PTP’s CloudOps and DevOps teams has been to fortify the validity of their research by streamlining and automating a mission critical data pipeline. The leading cloud infrastructure platform for Life Sciences, AWS, Terraform, and Nextflow, a software that is a recognized leader in scientific workflow systems, were chosen as cornerstone technologies to increase their likelihood for success.

After completing an AWS Well Architected Review, PTP had a solid working knowledge of how a couple of the client’s data pipelines had to work. PTP’s lead Solution Architect began using Nextflow to programmatically author a sequence of dependent compute steps tying multiple software apps (Cell Ranger, Seurat, Picard, and Star Aligner) to auto-configured AWS resources including E2, ELB, Auto Scaling, Lambda and Fargate. The integration of software tools and AWS infrastructure was completed, tested, and proven to work. From there EC2 Image Builder and Service Catalogs were set up to produce compute images in a controlled and repeatable manner. This allowed the client’s research teams to independently launch pipelines on AWS compute infrastructure via a pre-built Service Catalog. With this solution in place users can do their research securely online via a few easy-to-follow clicks. Governance of the work that these users perform is managed via AWS security policies with each user given the ability to launch relevant predetermined pipelines through the Service Catalog. The research process was, in a matter of weeks, automated, repeatable, adjustable and fully documented. This joint PTP and AWS solution ensures research validation while accelerating science. The image below demonstrates the overall environment.

All PTP builds have been put into Terraform templates to maintain known image files and component lists. Version control is handled by AWS Code Commit. As components change in Terraform, for example a software update to “version 4.2”, Terraform will know the file has changed and will deploy a new version of the component which then creates a new version of the recipe in Image Builder. For AWS cost optimization, which is extremely important when building infrastructure in this manner, the Service Catalog services are tied to Cloudwatch events. When devices go idle a trigger is pulled, then SQS queue and Lambda are used to terminate resources. This automates cost control. This systematic approach was arrived at by thoughtful engineering and has created an operationally efficient environment that can easily scale. The image below represents the Terraform and Service Catalog environment.

Recently, PTP has been working with the client to incorporate Amazon WorkSpaces for SAS access and AWS Managed AD. In the months ahead, PTP’s experience and consultative approach will be invaluable when determining the best options to isolate data and create additional levels of control and security.

PTP and the client’s IT and Informatics teams have implemented solutions that, at first, seemed complex and foreign. Embracing change is never easy, but by doing so the client has taken huge steps to avoid common data related pitfalls that face all Life Sciences companies, and thus has improved the likelihood of finding life changing relief for patients suffering from debilitating diseases.

Addressing Data Management Challenges in Life Sciences with Benchling and PTP CloudOps

Benchling is popular with scientists for its ability to leverage protocols to deliver a standard data format from the lab device and address FAIR data from the source.  However, there can be challenges for informatics because of the proprietary solution.  PTP and our CloudOps team is here to help!

The Challenge of Proprietary Lab Software

In the world of life sciences, data management can be a significant hurdle. Scientists often rely on proprietary software from various lab systems, which can lead to fragmented data management and compatibility issues. These challenges hinder the efficient extraction and processing of data from lab equipment, slowing down research progress.

Benchling: A Unified Solution for Lab Data

Benchling, a popular Electronic Lab Notebook (ELN) solution, addresses these challenges by providing a unified format for lab data. By establishing protocols to extract data from lab equipment, Benchling ensures that data conforms to FAIR (Findable, Accessible, Interoperable, Reusable) standards from the source. This standardized approach streamlines data management, making it easier for scientists to focus on their research.

Challenges with Data Extraction and Informatics

Despite the benefits Benchling offers, some informatics teams face challenges when extracting data for use in other applications or further analysis. These issues can include:

• Depth of Processing: The level of data processing and rendering may not meet the specific needs of the informatics team, leading to incomplete or suboptimal results.
• Data Extraction: Obtaining data from Benchling for use in other software or platforms can be complex, especially for custom analyses or integrations.
• Team Satisfaction: While scientists might find Benchling convenient, the informatics team may struggle with its limitations, affecting overall satisfaction with the data management process.

PTP CloudOps: Your Solution for Scientific Workloads

PTP, an AWS Life Sciences competency partner, offers a solution to these challenges through its Cloud Ops service. The PTP Cloud Ops team excels in automating and optimizing scientific workloads, providing the necessary tools and expertise to streamline data management.

With CloudOps, you can expect:

• Rapid Responses from Biotech-Focused Cloud Engineers: PTP’s team understands the unique needs of life sciences and offers quick support to address any issues.
• Automation and Optimization: PTP specializes in automating scientific workflows and optimizing data processing, ensuring that you get the most out of your lab data.
• Easy Onboarding and Purchase: Cloud Ops is available on the AWS Marketplace, making it easy to purchase and onboard.

Accelerate Your Science with PTP CloudOps

If you’re facing challenges with data extraction, processing, or overall data management, PTP Cloud Ops can help. By streamlining your scientific workloads and providing expert support, Cloud Ops allows your informatics team to work as effectively as your scientists.

Conclusion

Data management is a critical aspect of life sciences research, and proprietary software can complicate the process. Benchling offers a unified solution, but it may not address all the challenges faced by informatics teams. PTP Cloud Ops provides a comprehensive solution to these issues, enabling life sciences organizations to optimize their scientific workloads and accelerate their research. With Cloud Ops, you can improve data management, enhance team satisfaction, and focus on what matters most: advancing scientific discovery.

CloudOps Available on the AWS Marketplace!

One scenario that demands the use of Apache Airflow is managing workflows. Alvarez-Parmar and Maisagoni (2021) argue that as an open-source distributed workflow management platform, Airflow allows users to schedule, orchestrate, as well as monitor workflows. While using Airflow, one is able to orchestrate and automate complex data pipelines. Airflow runs on AWS Fargate alongside Amazon Elastic Container Service (ECS) as an orchestrator, implying that the user does not have to provision as well as manage servers. Airflow operates more than a batch processing platform since it allows the user to develop pipelines to process data and run different complex jobs in a distributed and complex manner. In relation to managing workforces, with AWS Fargate, a user can run the core components of Airflow without creating and managing servers. Similarly, one does not need to guess the capacity of the server to run the Airflow cluster or worry about autoscaling groups and bin packing to maximize resource utilization. Therefore, one practical situation requiring a user to run Airflow is when the user requires managing workflows. Alvarez-Parmar and Maisagoni (2021) note that “Managed Workflows is a managed orchestration service for Apache Airflow that makes it easy for data engineers and data scientists to execute data processing workflows on AWS” Airflow helps users orchestrate workflows as well as manage how they are executed without configuring, managing, and scaling the Airflow architecture. Users who run Airflow on AWS should consider Amazon Managed Workflows for Apache Airflow since it helps to set up Airflow, provisioning and autoscaling capacity (storage and compute), keeps Airflow up-to-date, and automates snapshots. Hence, in relation to using Airflow for managing workflows, Airflow provides a reliable and scalable framework for users to orchestrate data workflows. This enables data engineers to transform, extract, and load data from different sources. With Airflow’s operators, integration with various data systems like cloud storage, databases, and data warehouses is made easier.

In a different discussion on the practical use of Airflow to create on-demand or scheduled workflows that process complicated data from different data providers, Oliveira and Raditchkov (2019) argue that it becomes easier to orchestrate big data workflows with Apache Airflow. Large companies that run big data ETL workflows on the AWS work at a scale where most internal end-users are serviced and several concurrent pipelines are also serviced. With the continuous urge to extend and update the big data platform as a way of keeping up with the latest big data processing frameworks, what is required is an efficient architecture that simplifies big data platform management and enhances easy access to the big data applications. Therefore, using Airflow on AWS, centralized platform teams can maintain their big data platform, service various concurrent ETL workflows, and simplify operational tasks needed to attain the process. While managing workflows on AWS, the Airflow system uses Amazon EMR and Genie as open-source technologies. Amazon EMR provides a big data platform where workflow orchestration, execution, and authoring are done, while Genie offers a centralized REST API for effective big data job submission, central configuration management, dynamic job routing, and abstraction of Amazon EMR clusters. In the entire process of workflow management, Airflow offers a platform to enhance job orchestration, allowing the user to programmatically author, schedule, as well as monitor the complex data pipelines (Oliveira and Raditchkov, 2019). The role of Amazon EMR in the process is to offer a managed cluster platform that scales and runs Apache Spark and Hadoop, as well as other big data frameworks. The diagram shown below shows the use of Airflow in AWS to enhance big data workflows;

Airflow in AWS on big data Workflows Management (Oliveira and Raditchkov, 2019)

Other than using Airflow to support complex workflows, a different practical scenario that demands using Airflow in AWS is coordinating extract, transform, and load jobs. Airflow is used to perform ETL jobs because it works on a concept known as operators, which denote the logical blocks in ETL workflows (Sinha, 2021). To perform an airflow ETL job, a user is required to have an AWS account and Airflow installed in the system. Extract, transform, and load (ETL) job transforms raw data into needful datasets and finally to actionable insight. Anany (2018) argues that an ETL job reads data from various data sources and applies different transformations to the data before writing the results to a target where such data becomes ready for consumption. The ETL job targets and sources are relational databases in Amazon S3, which helps build a data lake in AWS. AWS provides AWS Glue as a service that deploys and authors ETL jobs. AWS Glue “is a fully managed extract, transform, and load service that makes it easy for customers to prepare and load their data for analytics.”

Despite the case, other AWS services that manage and implement ETL jobs entail the AWS Data Migration Service (DMS), Amazon Athena, and Amazon EMR. Therefore, in a practical scenario where one needs to orchestrate ETL jobs and workflow that entails different ETL technologies, with Airflow in AWS Glue and AWS DMS, the user will easily chain ETL jobs. A good scenario that would require the use of orchestration of ETL jobs using Airflow in AWS is when a business user wants to answer questions related to different datasets. In this case, if a user wants to find the correlation between forecasted sales revenue and online user engagement metrics like mobile users, website visits, or desktop users, the user will follow different ETL workflow steps, including the process of the sales dataset (PSD), Processing of the Marketing dataset (PMD), and Joining the marketing and Sales datasets (JMSD) (Anany, 2018). After doing so, the user will implement the ETL workflow using AWS Glue by chaining ETL jobs using job triggers. The diagram shown below demonstrates how the user will manage ETL workflow with AWS Glue based on the case scenario;

Using AWS Glue to solve the case scenario (Anany, 2018)

Despite the case, the following ETL architecture demonstrates how running Airflow on AWS to coordinate, extract, transform, and load jobs.

Running Airflow on AWS to coordinate ESL Jobs (Anany, 2018)

Finally, a different practical scenario that would necessitate a user to run Airflow on AWS is when the user wants to prepare and manage machine learning data and workflows. Cantu (2023) argues that Airflow efficiently helps manage machine learning workflows since it allows machine learning engineers and data scientists to schedule and define tasks for model training, data pre-processing, evaluation, and deployment. Hence, the ability of Airflow to schedule tasks and handle dependencies in a distributed framework positively impacts the management of the end-to-end lifecycle of the machine learning models. Machine learning workflows automate and orchestrate ML tasks’ sequences by allowing data transformation and collection. Training, evaluating, and testing the ML model to attain the intended outcome follows. Most customers use Airflow to perform scheduling, authoring, as well as monitoring of multi-stake workflows. AWS automates Amazon SageMaker tasks in an end-to-end workflow. Thus, one can automate their publishing datasets to Amazon S3, train ML model on their data, and deploy their model for prediction. Therefore, by running Airflow on AWS, one can easily “prepare data in AWS Glue before they train a model on Amazon SageMaker and then deploy the model to the production environment to make inference calls” (Thallam and Dominguez, 2019). Automating and orchestrating the tasks across several services makes it easier to build reproducible and repeatable Machine Learning workflows that can be shared among data scientists and engineers. When Airflow runs on AWS, preparation, and management of machine learning data and workflows are effectively done because AWS Step Functions monitors Amazon SageMaker to ensure that the jobs succeed. AWS uses different structures like built-in-error handling, state management, visual console, and parameter passing that monitor the user’s ML workflows to enhance success.

References

Alvarez-Parmar, R. and Maisagoni, C. (2021). Running Airflow on AWS Fargate. Amazon AWS. https://aws.amazon.com/blogs/containers/running-airflow-on-aws-fargate/

Anany, M. (2018). Orchestrate multiple ETL jobs using AWS Step Functions and AWS Lambda. Amazon AWS. https://aws.amazon.com/blogs/big-data/orchestrate-multiple-etl-jobs-using-aws-step-functions-and-aws-lambda/

Cantu, J. (2023). Mastering Workflow Management and Orchestration with Apache Airflow. Medium. https://medium.com/@jesus.cantu217/apache-airflow-a-comprehensive-guide-to-workflow-management-and-orchestration-bf1372e11920#:~:text=It%20provides%20a%20scalable%20and,cloud%20storage%2C%20and%20data%20warehouses.

Cantu, J. (2023). Mastering Workflow Management and Orchestration with Apache Airflow. Medium. https://medium.com/@jesus.cantu217/apache-airflow-a-comprehensive-guide-to-workflow-management-and-orchestration-bf1372e11920#:~:text=It%20provides%20a%20scalable%20and,cloud%20storage%2C%20and%20data%20warehouses.

Grzemski, S. (2020). Highly available Airflow cluster in Amazon AWS. Getindata. https://getindata.com/blog/highly-available-airflow-amazon-aws/

Oliveira, F. and Raditchkov, J. (2019). Orchestrate big data workflows with Apache Airflow, Genie, and Amazon EMR: Part 1. Amazon AWS https://aws.amazon.com/blogs/big-data/orchestrate-big-data-workflows-with-apache-airflow-genie-and-amazon-emr-part-1/

Sinha, v. (2021). Understanding Airflow ETL: 2 Easy Methods. Hevo https://hevodata.com/learn/airflow-etl-guide/#etljob

Thallam, R. and Dominguez, M. (2019). Build end-to-end machine learning workflows with Amazon SageMaker and Apache Airflow. Amazon AWS. https://aws.amazon.com/blogs/machine-learning/build-end-to-end-machine-learning-workflows-with-amazon-sagemaker-and-apache-airflow/

Ready to optimize your life sciences HPC pipeline?
Get a free cloud assessment to uncover inefficiencies, reduce spend, and align your cloud architecture with research goals.

One of the numerous advantages we have within the PTP PeakPlus™ team is our ability to customize, automate, and expand security services that, in some cases, even the security vendor is unable to provide.

Our customer needed a replacement for Cisco Cloud Web Security (CWS) which they used for web proxying, URL filtering (keeping users from going to the wrong sites), and user internet usage reports (finding people that go to the wrong sites). The obvious and recommended choice from Cisco is the awesome Umbrella solution. It provides everything the customer needed as well as enhanced DNS and web security across their organization. The only issue was that for compliance the customer needed to be able to provide 90 days of log retention vs the standard 30 day limit the Umbrella product provides. Cisco does provide the ability to store the logs off the system, but this would mean the customer would have to be able to sift through raw logs then take that data and put it into some sort of readable format. This all equals a NO GO from the customer point of view.

So, the customer asked, “Can we fix it?” PTP answered, “Yes, we can!” (Is this a Bob the Builder reference?)

In order to meet the expectations of the customer, PTP provided a single, easy-to-use platform for user internet reporting which enabled search functions with adjustable date ranges, and easily digestible reporting clearly displaying user names, websites visited, and time/date.

Now if you want to get a little more into the weeds…

To get the user internet usage logs out of Umbrella we leveraged our strategic partner, AWS, and their S3 service. A new storage bucket and access policy were created in AWS to allow Umbrella to store compressed proxy and DNS logs in our corporate S3 buckets. Data lifecycle policies (within AWS S3) could then be deployed to maintain data for the required 90 days. In this format the data is still not searchable or useful to the customer.

Next, the user internet usage log data is moved out of AWS S3 and into AWS RDS to provide the required search functions needed for displaying the data in a readable format. The Security Services team created Lambda functions in AWS to move the data into RDS. These custom functions were set up with triggers allowing them to be run every time a new file added to AWS S3 bucket.

Custom scripts then decompress the data, normalize it, de-duplicate it, and index it into an AWS RDS instance running MariaDB. Millions of rows of data are currently being processed by these Lambda functions every day for this customer.

The final piece provides a place to search, view and print reports of the data from our PeakPlus View customer portal. This was achieved through new reports and integrations created to fetch and display internet usage data.

The customer is now able use the native functions of Cisco Umbrella while enjoying enhanced functionality due to the PTP Security Services team’s ingenuity and tremendous work by PTP’s engineering team.

Secure cloud automation helps meet compliance where native tools fall short.

PTP extends AWS and Cisco capabilities to meet your exact needs with DevOps-driven security solutions.