Web Scraping for Pharmacovigilance

Post 71 scrap 1 1

Pharmacovigilance concept

Including web data extraction techniques that capture data from publications of patients and users of social networks, new information on adverse effects derived from the administration of medications can be processed. These techniques are being fundamental support for decision-making of the pharmaceutical industry.

How is it possible to improve drug risk monitoring based on the advances in web scraping and the automation proposed by digital data extraction from the web? How will pharmaceutical laboratories and regulatory organizations be able to cope with the data sets that arise from the processing of drugs and enhance their exploitation to make them safer every day?

When a new drug is about to be launched for commercialization, it is authorized on the premise that the benefits of its administration outweigh the latent risks that it poses for the target population. However, not all potential or actual adverse reactions are identified at the time their initial market release is authorized. At this point, Pharmacovigilance acquires a central role in the public health system: it is the continuous process of detection, evaluation, analysis and prevention of adverse events resulting from the administration of drugs or other problems related to them (errors, falsification, lack of efficacy, misuse or abuse, etc.) both in its pre-commercial phase and during its subsequent approval and commercialization.

Pharmacovigilance history

The concept of Pharmacovigilance began to be used in the 1960s in Germany from the effects produced by Thalidomide, which had been launched on the market in those years. This resulted in the governments of the different countries as well as the global health organizations will initiate a stricter control to increase the safety of the patient and the population.

Pharmacovigilance Data Extraction

Usually both in our region and in developed countries Pharmacovigilance systems are imperfect: notifications are often redundant on known effects, information is scattered or poorly structured, there is a lack of motivation of professionals to comply with notifications, there are inequitable health systems, little interaction between patients and professionals, as well as the supply of drugs with unapproved indications. Beyond these problems, the intensive use of data and traditional monitoring is increasingly necessary by the industry, periodically requiring historical information of the complete life-cycle of the drug: from conception, trial and approval to the events reported after its administration in the market. 

It is imperative to implement active programs based on precise, centralized and systematized information, which complement the traditional methods of notifications and databases of public organizations. Can digital multi-source processing technologies and web scraping help improve drug risk monitoring, making these processes more profitable for the companies that produce and market them? Definitely.

Pharmacovigilance for digital transformation

Given the exponential growth in the use of the web for health issues – including health information websites, medical newsletters, discussion forums, blogs and social networks – the complexity and domains where pharmacovigilance systems can obtain data have multiplied. 

At the same time, health information published online by patients represents a potentially valuable source of information for both pharmaceutical companies and public health organizations. In the last decade, numerous Social Networking Sites (SNS) and Web Applications have appeared and proliferated, a product of the so-called “digital transformation”. Those sites dedicated to health communities have become popular, being digital spaces that allow their users to exchange information about their health condition with others who deal with the same problems or pathologies and receive peer support. These sites include categories such as: 

a) Public online platforms: they host a large number of health-related communities and groups, and also contain large volumes of individual user posts related to health issues (eg. Facebook, Twitter, Tumblr, etc.). 

b) Specialty Healthcare Social Media and Forums: generic networking sites on general health and disease support topics, typically requiring user profiles (www.patientslikeme.com, www.dailystrength.org, www.medhelp.org, https://exchanges.webmd.com, http://curetogether.com/), where users discuss their health-related experiences, including prescription drug use, side effects, and treatments. 

c) Exchange platforms focused on medicine: patient forums, which allow sharing and comparing experiences in the use of certain medications (http://www.askapatient.com, http://www.medications.com/). There are also more specific online health forums, focused on diseases.

Pharmcovigilance Data Extraction

Pharmacovigilance Conclusion 

There is no doubt that these data and opinions – published spontaneously by users – are one of the potential sources with the greatest added value in the area of ​​pharmacovigilance. However, the development of sophisticated algorithms that can deal with this complex and enormous volume of heterogeneous information coming from social networks is a very incipient open work. There is still a long way to go to continue exploiting these web data extraction techniques, so that they can contribute to optimizing decision, making both in the pharmaceutical industry and in the regulatory organizations of the activity.


No matter your data needs, we can help.

We identify, extract, clean, filter and deliver the data in the desired format ready for use in your database or in your upload queue. We adapt the data delivery to your integration requirements. We provide a reliable, secure, robust, and traceable response to your web-data need. We have the expertise to solve highly complex extraction tasks: OCRs, multiple-steps-extraction, proxy management, etc. We provide our 7x24 web-data storage and processing infrastructure, SLA 99.999%

    We’ve helped over hundreds of companies with needs of scrapers. Ready to know how we can help you?

    Follow Us