Adding a UI to a migration script

Shellviz makes it easy to add UIs to your migration scripts with almost no additional overhead. You can track the progress of migration scripts in real time, get detailed insights into the data that your migration script is outputting, and spit out multiple logs in easy-to-explore outputs.

Scenario: You recently added a company field to your user model, and users have slowly been entering their company values using our UI. You decide to write a data migration that pre-populates company values for existing users based on data you have from your CRM. This is a big migration, and you would like to be able to visualize and store the affected users.

Note: The code in this scenario is presented as Python, but everything shown here can also be done using Shellviz' Javascript library. Refer to the Docs for implementation details on each library

Getting started

For the sake of this demo, let's say your migration looks a little like this

num_users = len(users)
for index, user in enumerate(users):
	company = user_company_dict[user.email]
	if user.company != company:
		user.company = company
		user.save()
	print(f'Updated user {index} of {num_users}')

Track the progress

Shellviz's Progress visualization provides a nice UI for tracking the progress of the migration. Let's start by initializing Shellviz, then adding a progress visualization

from shellviz import Shellviz
sv = Shellviz()
sv.progress(0.0, id='progress')

Note the id value! This is key, as it will allow us to update the value of a previously initialized visualization

Now, let's add the progress tracker to the migration script

num_users = len(users)
for index, user in enumerate(users):
	sv.progress(index / num_users, id='progress')
	# ...
	print(f'Updated user {index} of {num_users}')

You will now see a progress bar that shows the progression from 0% to 100% as your migration script runs

Output a log of changes

It would be nice to show a log of results in addition to the progress tracker. Let's do this now!

num_users = len(users)
log = ''
for index, user in enumerate(users):
	sv.progress(index / num_users, id='progress')
	# ...
	log += f'Updated user {user.email}'s company to {user.company}\n'
	sv.text(log, id='log')

Note: Shellviz does not currently support appending data to an existing visualization, but this is something we plan on adding soon!

Outputting the log as a table

Now let's take advantage of the additional visualization options available to us with Shellviz. Rather than output logs as text, we can present the data as a table.

num_users = len(users)
updated_users = []
for index, user in enumerate(users):
	sv.progress(index / num_users, id='progress')
	# ...
	updated_users.append({'email': user.email, 'company': user.company})
	sv.table(updated_users, id='updated_users')

We show the results of the migration in a table, allowing us to more understand the data, filter it, sort it, or download it as a CSV file.

Outputting multiple logs

num_users = len(users)
updated_users = []
unchanged_users = []
for index, user in enumerate(users):
	sv.progress(index / num_users, id='progress')
	# ...
	if user.company != company:
		unchanged_users.append({'email': user.email, 'company': user.company})
	else:
		updated_users.append({'email': user.email, 'company': user.company})
	sv.table(unchanged_users, id='unchanged_users')
	sv.table(updated_users, id='updated_users')

Note: It may make more sense to have one table with a status column in it, but this example is designed to show how multiple data values can be output at a time from one migration file.

Summary

We've added useful visualization and logging capabilities to our migration file with only a handful of lines of code:

from shellviz import Shellviz
sv = Shellviz()
sv.progress(index / num_users, id='progress')
sv.table(unchanged_users, id='unchanged_users')
sv.table(updated_users, id='updated_users')

Performance Note

In its current state, Shellviz does not handle large numbers of requests optimially. You may therefore find that adding this code to a large migration script may cause your script to slow down or for you to run into rate limits. We have plans to add batch updates, but for the time being you may want to batch your updates in your migration script:

if index % 10 == 0 or (index + 1) == num_users:
	# only send updates every 10 users or when we reach the end of the migration
	sv.progress(index / num_users, id='progress')
	sv.table(unchanged_users, id='unchanged_users')
	sv.table(updated_users, id='updated_users')
	sv.table(updated_users, id='updated_users')