Instant Gratification? The importance of time when handling scholarly data

Damaris Murry, Robert Nelson, Richard Outten

Keywords: ETL, system performance, user engagement

TL;DR: The importance of shielding end users from as many complexities of the ETL process as possible.

Abstract:

As we engage with users of Scholars@Duke (VIVO), we understand that there are particular dates and times where engagement with the system and the underlying source systems is at a peak. We have also experienced how these peak usage times can be taxing on the systems, the profile owners, and the people that support them! And while there is almost always a logical reason for why something doesn't work (or the data doesn't look) quite as expected, this presentation is about the importance of trying to anticipate user needs and questions and shielding them from as many complexities of the ETL process as possible.

In this talk, we will present some of the key features we've put in place, such as periodic data integrity checks, timestamps, status notifications, on-demand refreshes, and widespread messaging that have been key to our success with profile owners.