In running operations one of the activities that you often have to take on is testing. In larger environments where we are dealing with dozens if not hundreds of different systems all needing to work together to make different use cases achievable is not always an easy thing to-do. Typically there will be a “golden copy” of the data set that has everything anonymized, all the row’s columns with just the right values for the tests to be prepped and ran through all of these different systems.
You also have business users, developers, data engineers and data scientists that also need this “golden copy” too to-do their day to day jobs often enough. Now folks are storing a backup and restoring it all over the place and managing all of the databases that are running all over the place. When you have multiple systems to restore too and re-configure it also often makes it harder.
When running a Lakehouse things are no different in complexity in user needs but the solution is better with the right catalog. So lets jump in and take a look at Project Nessie: Transactional Catalog for Data Lakes with Git-like semantics.
So now what you can do in the UAT and below environments is create one big data lake. Create all the tables and fill it up with all the anonymized data you want and get all the values just right. Then create a branch for the “golden copy” which you can do off of a commit hash and capture the EXACT data change you want. Then each time you want another person or project or whatever have access to the “golden copy” that they can write to and test and see if what they are doing is working you have a system already built into your data lake. The commands are easy once everything is running.
nessie branch -o 8e2d048ca golden_copy_v1 main
nessie branch uat_week7_test_32 golden_copy_v1Now in your config you can set spark.sql.catalog.nessie.ref to uat_week7_test_32 and your data set will start with what was in golden_copy_v1 but your writes all will only show up in uat_week7_test_32 and no other ref will be touched.
Besides commit hash you can also branch off of timestamp and take advantage of the time travel features but now instead of just reading from the data set you can now also write to what you are reading and those writes will only affect your branch.
I updated the source code that I did for the Getting Started with Apache Spark, Apache Kafka and Apache Iceberg to now include Nessie integrated to all the scripts and docker compose so everything works with Nessie now and you can try it out. If you want to see the specific changes go to the first pull request and check it out. Once you have docker-compose up running you can go to the UI http://localhost:19120/ or use the CLI.
Thanx =8^) Joe Stein
https://www.twitter.com/charmalloc
https://www.linkedin.com/in/charmalloc
Leave a Reply