A frequent inquiry from customers goes something like this: "I ran a query on the [data warehouse or SQL analytics endpoint] and now I want to know how much that query cost me."

On the surface that seems easy enough. Afterall, the Capacity Metrics app tracks the usage for every operation that occurs on your Fabric capacity right down to the individual T-SQL query.

Before getting into how to get the capacity usage, let's outline the assumptions this post is going to make:

1. You understand the association between workspaces and capacities.

2. You understand the concept of smoothing as it relates to capacity usage.

How would I do this manually?

However, when you try to use the app, you soon realize it's far more cumbersome than you'd like. This is the process you need to go through:

1. Run a query using the SQL endpoint (web browser, SSMS, ADS, etc.).

2. Gather the distributed_statement_id and query execution date and time from Query Insights (or capture it through another method of your choosing).

3. In the capacity metrics app, navigate to a time point that is at least 15 minutes after the query finished running.

4. Filter the background operations table to just the operations you want to see (Operation Id = distributed_statement_id).

It's step number 3 and 4 that cause the most pain and what we will address here today programmatically. Step 3 is just confusing for people because you need to wait for the query to show up in the report, but you need to go to "someplace after the query finishes running", usually 10-15 minutes, and find the query there. Step 4 is the bigger challenge because you need to get the list of IDs in step 2 then search for one of them in the filter in step 4, click the check the box, go back to step 2 to copy the next ID, search for it, and repeat that over for each query's distributed statement id.

The process is cumbersome for one query, it will drain your evening if you need to do this for a lot of queries, and it's just impossible if you need to do this for any number of queries at scale.

How would I do this programmatically?

The overall process is similar: Run a query (or queries), get the distributed_statement_id list, look them up in the capacity metrics app.

To accomplish this programmatically we will need to make a couple of changes to the manual process above. Let's do an overview then we'll dive into details.

1. Run a query using the SQL endpoint (web browser, SSMS, ADS, etc.).

2. Gather the distributed_statement_id and query execution date and time from Query Insights. Turn it into a string in the format of "distributed_statement_id_01", "distributed_statement_id_02", "distributed_statement_id_NN"

3. Enter parameters into data engineering notebook.

4. Run the notebook.

Before you try to use this method, you will need to download the notebook Get SQL query CUs from Capacity Metrics which is hosted on my GitHub account. Then, upload the notebook to a Fabric workspace. The workspace will need capacity to run the notebook. The notebook can be in any Fabric workspace; it does not need to be collocated with the Capacity Metrics app or with the workspace where your lakehouses/warehouses live. In fact, this notebook doesn't even require a lakehouse to be in the workspace to run.

Let's look at a walkthrough of this.

Run queries

I think this is self explanatory. Run your queries however you'd like. Keep in mind that it takes several minutes for the queries to show up in Query Insights and a few more minutes to show up in the Capacity Metrics data. Just assume that you need to wait at least 15 minutes after running a query for this whole process to work (the same is true of the manual process above as we are using the same dataset).

Gather the distributed_statement_id(s)

Here is where things deviate from the manual process. While we are still going to be using Query Insights, we want to combine all the distributed statement ids into a single string, grouped by the date the query was run. This is necessary for the magic of a solution that allows you to get the CUs in bulk. Consider the following query to be a guide on how to do this but by no means the only way. The key is the format: double quotes around each distributed_statement_id forming a comma separated list.

SELECT
    CONVERT(DATE, start_time) ExecutionDate,
    COUNT(*) AS CountOfQueries,
    '"' + STRING_AGG(CONVERT(VARCHAR(MAX), distributed_statement_id), '", "') + '"' AS OperationIDList
FROM
    (
        SELECT
            *
        FROM queryinsights.exec_requests_history
        WHERE
            command LIKE '%%'
            AND start_time >= '2024-05-08 20:53:38.000000'
            AND start_time <= '2024-05-13 20:25:53.000000'
    ) AS queryinsights_requests
GROUP BY
    CONVERT(DATE, start_time)

The output from this query will provide the date the queries were executed, how many matched the subquery criteria, and the complete string of query ids. You will want to copy the ExecutionDate and the corresponding OperationIDList for the next step.

For the screenshots later in this post, I chose the 14 queries that were run on May 13th.

Enter parameters in the notebook

The big departure from the manual process is that we will use a notebook to query the Capacity Metrics dataset to get the data we need. You will need to import the notebook to any Fabric workspace (it requires capacity to use Spark) then fill in the 5 parameters in the first code cell.

• capacity_metrics_workspace

  • The name of the workspace that hosts the Capacity Metrics app.

• capacity_metrics_dataset

  • The name of the dataset used by the Capacity Metrics app.

• capacity_id

  • The capacity which hosted the workspace where user query were run.

• date_of_operations

  • This is the ExecutionDate column from the prior step.

  • The date the operations were run.

• operation_id_list

  • This is the OperationIDList column from the prior step.

  • The list of operations of which you want to collect the capacity unit usage.

  • This should be a comma separate list and each operation should be enclosed in double quotes

  • For example: "AAAAAAAA-BBBB-CCCC-DDDD-1234567890AB", "EEEEEEEE-FFFF-GGGG-HHHH-1234567890AB"

Run the notebook

That's it. Run it!

After the notebook runs, scroll down to the very last cell where the dataframe will display each distributed_statement_id (this is the OperationID field in Capacity Metrics) the total CUs consumed by that query and some additional information.

Here you can see the total CUs for each of the 14 queries from step 2.

Did you also notice that we didn't have to deal with a timepoint? We only had to know what date the SQL query ran. You didn't even have to know the time that the query ran. The amount of time that I spend waiting for time period to show up in the capacity metrics report just so I can search for my data is wild. And it's all gone!

The magic that makes this happen is Semantic Link which provides a bridge between the data engineering and Power BI experiences in Fabric. It contains a variety of functionality including refreshing semantic models, reading tables from a semantic model, listing workspaces, getting workspace information, and executing DAX statements to return a dataset from a semantic model which is what this notebook uses.

Wrapping it up

There you have it! Now instead of looking up each query individually, you can just run a simple notebook to look up the CU usage in bulk! Let's wrap up with some key things to be aware of.

1. Here is a link to get a copy of the notebook (Get SQL CUs from Capacity Metrics) from my GitHub account.

2. You still need Capacity Metrics. Make note of the workspace name and semantic model name. Those are needed.

3. This notebook assumes that your Capacity Metrics semantic model is refreshed. Perhaps I will extend this in the future to add model refresh.

4. You will need the ID for the capacity that runs your workspace, not just the capacity name. This is the one part that's a little more difficult than the manual approach.

5. You do need to run this notebook for each date individually. I wanted to make this as easy as possible to use, so I started simple. It's still far faster than looking up individual queries manually. The nice thing though, is you can take this notebook and wrap it in another process that can be much more flexible.

   For me, I have another version of this project where I persist the data from queryinsights.exec_requests_history into a user table, run this notebook to get the CU information, and then update the user table with the CU usage. The next time the process runs it only looks up the queries from query insights that do NOT have any CU usage already.

   You could also do this across many workspaces to centralize the data. You could parameterize this with a pipeline instead of calling it from another notebook. The possibilities are endless! Would love to hear what you have built with it though!

6. I believe there is a limit of 10,000 or 100,000 records that can be returned at any one time. Yes, those numbers are very different. I don't know this for a fact though, but I'm fairly certain that limit is in place. Which means if you have a highly used system, you may need to break the query list into chunks and feed them through. Another great reason to parameterize this and build a framework around it.

What do you think? Time saver? Did I waste my time building this?

Let me know in the comments what you think!

Which leads me to today's question...how do I get a list of all the parameters for a Python function? In today's case: mssparkutils.credentials.getSecret.

You could go to the documentation, but that's often not complete (I'm looking at you, my fellow Microsoft employees on the documentation team!). Usually, I just want to know what my possibilities are so I can at least throw some options in there to see what happens. Maybe I'll get lucky. That's where inspect.signature comes into play.

help

The first thing I could try is using Python's built in "help" method. By simply passing a module, method, function, keyword, or other object the help information from PyDoc.

Python

help(mssparkutils.credentials.getSecret)

Easy enough. What if I want a little more detail?

inspect

The inspect module can help get some useful information for us in this case as well. Using my new friend help, I can see this will give me a look at useful information from Python objects. Here we will look at two functions: inspect.signature() and inspect.getfullargspec().

To begin, we will import the inspect module.

inspect.signature()

As you can see, running inspect.signature(msparkutils.credentials.getSecret) returns what I need in a simple output which is very similar to what we say with help(mssparkutils.credentials.getSecret) earlier. It gives me the list of inputs which are akvName, secret, and linkedService which is optional as it has a default value assigned to it.

But I am a simple person. I need things spelled out a bit more explicitly sometimes.

inspect.getfullargspec()

When I run inspect.getfullargspec(mssparkutils.credentials.getSecret) I am given a much more detailed breakdown of the output. While help and signature() will usually get the job done, it is nice knowing that I can get a full breakdown of the inputs and outputs.

I am sure there are reasons to use signature over getfullargspec and reasons to use getfullargspec over signature. I saw something about a decorated function being handled properly by signature but not getfullargspec. Since the only possible time I can think of to use a decorator function is when I'm putting up Christmas decor and it is currently May, I think that is a deep dive topic for future Brad.

As you may (or may not) know, I work on the Fabric product team at Microsoft. I am a principal program manager on the Fabric CAT (Customer Advisory Team) team. I always feel strange saying CAT team because then it's team team. It's like trying to explain Microsoft Teams to people. "Just post it on our Teams...uh...team?" or "I need to go to the ATM machine to get cash." You see cash is what people used to use to pay for things before...ok, we are wayyyy off topic. Where was I?

Ah yes, I work on the Fabric CAT team, and I often go talk to people about Fabric as a whole. However, most of the demos I do are just touching on one piece of the puzzle. What I'm setting out to do with this series is show the process of building an end-to-end solution in Microsoft Fabric from data source to visualization showing various options along the way. My goal is to be able to walk into any room and be able to say "let me pull up a workspace a show you what this looks like" for most of the major Fabric functionality.

Let's cover some of the basics...

What this will not be

Let's get this out of the way first. What will this demo not be, or what purpose will it not be designed to serve.

• This will not be comprehensive of every single piece of functionality in Microsoft Fabric. We are sticking to major themes, the Heading 1 topics if you will.

• This will not include every experience, just the ones I deal with on a regular basis. Think data science or private endpoints. Maybe it will expand in the future, but we'll see where it goes. The idea is big picture, make the concepts simple.

• This will not be complicated where it can be simple. Again, let's not boil the ocean. We aren't hitting every security feature. I'll leave that for individual posts or other people to cover.

• This will not be smoke and mirrors. I will only use what everyone has access to, and I won't hide any code from you to make things look easier than they are or fake any data along the way.

• This will not show every architecture option. I honestly don't know if I am going to show loading data using Dataflows Gen2 since I can do everything I need in a pipeline copy activity. Again, we shall see what happens. Maybe I'll have secondary posts that cover alternative methods or design patterns.

What this will be

Now that we have some ground rules for what this project will not be, let's go over what it will be.

• This will walk you through every step of building an end-to-end analytics solution on Microsoft Fabric.

• This will be something you can build in your environment as well. I'll make everything available in one form or another, be it screenshots, step-by-step clicks instructions, code on GitHub, or videos on YouTube.

• This will be using the Microsoft Wide World Importers dataset. It's easy to understand (retail sales), has all the translation code to go from the OLTP to DW, and is easily available to everyone.

• This will be as representative of the real world as possible while still keeping the concepts simple. For example, we will use an Azure SQL DB for the source data, but we aren't going to be simulating transactions on the source system and build slowly changing dimension logic. It's easy enough to extend the solution to include those items if you really want to.

• This will be built out over the course of N weeks. That is not a typo, that's not a placeholder I forgot to update, that is the truth. I don't know how long this is going to take. I've thought about doing one post a week, breaking up into approximately 30 minute building sessions, covering one step at a time no matter how long it takes. This is going to change as we go along, I just don't know how long it's going to take. I know I can build everything from scratch in a few hours, but teaching/blogging time is a different animal.

• This will be somewhat sanitized but, as I mentioned above, I won't hide things from you. What I mean by that is that to covert the Wide World Importers into Wide World Importers DW there are a series of views and stored procedures that need to be built/converted. I'm not going to cover every step of that code building/conversion process because it's frankly a bit boring and doesn't serve the purpose of this series, but I will give you all the real code. That means some of the steps you have to do in a real-world project will be happening behind the scenes or will be already done for you. Again, our focus is Fabric not project management or BI development.

What the architecture will look like

This is still to be defined, but here's what I'm thinking:

Azure SQL Database --- Data Factory Pipeline ---> Lakehouse Files --- Notebook ---> Lakehouse Tables --- T-SQL ---> Data Warehouse --- DirectLake ---> Power BI report

Be on the lookout for the first post in the series where we will setup the source database soon!

I hope to see you there!

Date: Wednesday July 19, 2023
Time: 6:00 - 8:00 PM Eastern
Where: Keiser University (6430 Southpoint Parkway Suite #100, Jacksonville, FL 32216
Cost: Free!
Registration: Meetup

Microsoft Fabric 2023 Year in Review and 2024 Roadmap

Less than one year ago Microsoft Fabric was announced at the Build 2023 conference. Since then the team has been hard at work rolling out new features and functionality each month. From small improvements like the ability to convert entire folders to Delta with a right-click rather than going file by file or background improvements like faster CSV loading to marquee functionality like the launch of multiple Copilots and the announcement of general availability or the new database mirroring, 2023 was a huge year for Fabric. We will run down the list of some of the biggest announcements and show demos along the way so you can see the exciting functionality that is being delivered each and every month. Then, we will take a look at the year ahead with a sneak peak into the 2024 roadmap. If you're using Fabric or considering using Fabric for a project, you'll want to join this session to see what all the buzz is about!

As the Christmas season closes and we move deeper into winter there is now a new year to look forward to which means it's time for new year's resolutions.

Resolution

A firm decision to do or not to do something.

I'm not a big fan of resolutions. They are often lofty, unattainable (realistically anyway), and leave little room for error which means people abandon them as soon as things start to go off the rails even a little. Saying "I resolve to go to the gym 3 times a week" psychologically gives you an out as soon as you miss that third day. It creates the feeling of "I said I was going to go this, I messed it up, now my streak is done, and I don't need to keep trying." In fact, almost 25% of people give up on their resolutions in the first week, and more than 2/3 give up before the end of January.

Goal

The object of a person's ambition or effort; an aim or desired result.

Instead, I like to set goals. Goals give you something to aim for but allow for the inevitable error and adjustment that comes along with being human and needing to respond to life events. Someone with the goal of exercising 3 times a week that has a lot of travel coming up would have the freedom to exchange the gym with a jog around the park or a workout in their hotel room before an early morning flight. And if they don't hit the goal that week, then they can evaluate what went wrong, how they can do better and get back to it the next week. Or if they set a goal of 5 days at the gym, they may have a baby and realize that's no longer viable and adjust to 2 gym days with walks on the other days.

Are those two things really all that different. No, of course not. But the mindset that comes with resolutions vs. goals tends to be very different.

What then are my goals for the new year? Some of these are aimed at being healthier so I can live a long, fulfilling life for my family. Some of these are centered around being more intentional with my time. And likely the most important are about deepening my relationship with Jesus Christ. Some of these I already do and want to make sure I don't lose focus, some are things I don't do well and want to improve, and others are areas I haven't started yet.

1. Read through the entire Bible this year (The Bible Recap)

2. Pray every day with the family and on my own

3. Listen to 1 audiobook each month (I'm not a big reader, but I'll listen!)

4. Dedicate as much time between 5:00 PM and the boys bedtime to my family as possible (less screen time/work and more play time with the kids and talk with Nichole)

5. Spend more quality time with Nichole after the boys go to bed (puzzles, crafts, just chatting)

6. Get more sleep (fewer work until 2:00 - 3:00 AM nights)

7. Daily walk and/or bike ride with the family

8. Do something outside with the family every weekend (bike trail, kayak, beach, park trip, etc.)

9. Eat better (less Dr Pepper, more meals at home, fewer carbs, more veggies, all that good stuff)

10. Reduce and shift the kind of content I consume (more history, Bible commentary, educational material and fewer TV shows/movies)

11. Create more content than I did last year (more blogs, more videos)

What do you think?
Am I nitpicking about settings resolutions vs. goals?
What are you resolving to do this year?
What goals are you setting for yourself?

I hope to see you there!

Date: Thursday July 20, 2023
Time: 7:00 - 9:00 PM Eastern
Where: Online (Register to receive the link)
Cost: Free!
Registration: Meetup

Introducing Microsoft Fabric: The All-in-One Analytics Solution

The world is awash with data and it is up to data professionals to make sense of it all. Just a few weeks ago Microsoft announced the next generation of analytics at Build: Microsoft Fabric!

Fabric is a complete end-to-end analytics platform bringing together Azure Data Factory, Azure Synapse, and Power BI in a single location and more deeply integrated than ever. It’s built on a SaaS platform which helps deliver innovation more quickly and allowing users to get up and running in seconds. At the core Fabric provides a lake-centric and open data hub using the popular Delta format with built-in security, governance, and compliance throughout. At the edge, Fabric delivers flexibility for data scientists, data warehouse developers, and Power BI users to build and analyze using their unique skill sets.

During this session we will discuss the guiding principles and architecture behind Fabric and show a lot of demos! Whether you’re a data engineer, data scientist, data warehouse developer, or an existing Power BI user, Fabric has something for everyone. You won’t want to miss this session!

On the surface Fabric looks like an iteration on the work that was done when Azure Synapse Analytics was launched a few years ago. The reality is that this is not just a step forward but a leap. Many of the components of Fabric do look similar to Synapse, including carrying forward the brand in several of the workloads like Synapse Data Warehousing. However, the architecture, integration, and delivery are vastly different.

OneLake

We have to start with OneLake because that is the thread that holds this Fabric together. Everyone is going to write, Tweet, or say that so I'm just going to get it out of the way. Please don't hold it against me. I know, low hanging fruit and all, but it is what it is. Moving on... At the core, OneLake is Azure Data Lake Storage but it's so much more than that. Gone are the days of provisioning storage accounts, setting up private endpoints between storage and compute, copying data so different compute engines can access it, managing security in multiple locations, duplicating data when you want to create development environments, and so much more.

You don't create OneLake. You just have OneLake. It's described as OneDrive for your data, which now that I think about it, I already store data in OneDrive...I'll figure that all out another time. The point is, you interact with OneLake just as easily as you interact with OneDrive. It's driven by your Azure AD account by default. It's easy to share with others. You don't have to worry about how to tune to get the necessary IOPS. It's just there.

The best part is that all of the compute engines in Fabric store their data in OneLake, which means all the compute engines can access each other's data. All the architects out there are probably thinking "Does this mean I don't have to do all that data copying to give my SQL people access to the data in my lake? And does that mean I don't have to copy the data from my warehouse to a dataset in Power BI to be able to report on it?" That is absolutely correct. Along with OneLake comes "One Copy" which means every engine can see and interact with the all the data (with proper security of course).

The other best part...or maybe the equally best part, however you'd say that...is OneLake acts just like Azure Data Lake Storage. That means my data is not locked inside OneLake never to be seen again by any of the other Azure services. Therefore, all the Databricks users out there can read and write data to OneLake meaning you can continue to leverage all of those investments and integrations with other services that you've spent years building.

The other, other best part (this is getting out of hand) is if you have an existing Azure Data Lake Storage account you can create a shortcut and bring that data right into OneLake without even having to move it! So everything in your data estate can then be seen and analyzed through OneLake. More on all of this in a future post. As you can see, there is a huge amount of integration that OneLake unlocks and it's arguably the most important piece of the entire puzzle.

One format

In addition to OneLake there is one other key change that enables all the functionality in Fabric: standardizing on Delta. Every compute engine in Fabric now reads and writes Delta format. This is how we prevent the need to copy data from the lakehouse into the warehouse or into the lakehouse from Kusto. Why Delta? At this point, it's the industry standard. Sure, there is some secret sauce going on under the covers that enables all kinds of nifty functionality but that's for another time...

Workloads

Fabric supports a variety of workloads that are more deeply integrated than ever before. We will go into details of each of these workloads in separate posts, but for now let's talk about what is built into the platform.

• Data Factory - Easily integrate and transform data from any source. Provides no-code and low-code transformation experiences and continues to be the orchestration hub of the data solution.

• Synapse Data Engineering - Provides data engineers a familiar, notebook-based experience for transforming data at scale using Spark.

• Synapse Data Science - The place to go for managing, training, and deploying machine learning models.

• Synapse Data Warehousing - A relational data warehouse with truly independent compute and storage that provides industry-leading performance. This is NOT Synapse dedicated SQL pools and it is NOT Synapse serverless SQL pools.

• Synapse Real Time Analytics - Dive into large volumes of data from apps, websites, IoT devices and any other time series data you can get your hands on.

• Power BI - I feel like this doesn't even need an introduction. Create interactive reports with stunning visuals (I'm not that creative, but I know some people that are) to gain insights from data across your organization.

• Data Activator - Coming soon! - A detection system that provides alerting and monitoring, so you know exactly when important information is changing.

Experiences

The experience in Fabric line up with the workloads to reduce the noise and quickly surface the most relevant information for what you need to get done.

Selecting an experience, such as Synapse Data Warehouse, will bring you to a screen that shows common tasks and resources that a user working on a data warehouse would find useful. In this case, the ability to create a new warehouse, create a new data pipeline, or link directly to the getting started documentation.

Software-as-a-Service

Fabric will be delivered like no analytics platform at Microsoft before. Fabric is leveraging a Software as a Service (SaaS) model. This doesn't mean that you have any less functionality, you will still have complete control over your data and experience like you have in all Azure PaaS services. However, it does mean some things are changing for the better including:

• It just works.

• One capacity that will be shared between all the workloads. No more paying for each individual dedicated SQL pool and worrying about paying for dedicated capacity in multiple databases that may sit unused.

• Updates delivered weekly! That means the platform can be updated with the latest and greatest functionality all the time rather than waiting for less-frequent "big bang" releases.

• Trials to get started in seconds if you just want to kick the tires.

• Fast provisioning and automatic scaling. A data warehouse takes about 10-20 seconds to spin up rather than the 10+ minutes that it takes today. Spark pools come online in less than 15 seconds rather than 5+ minutes today.

• Success by default, meaning fewer knobs to tune because the best practices are implemented automatically (for SQL think things like stats always being up to date) and the workloads are all integrated seamlessly.

Wrapping up

This is the biggest advance in analytics at Microsoft since I started building data warehouses on SQL Server back in 2009. Full disclosure, I may be a little biased since I work on the Fabric product team, as of the time of writing this post I am a member of Fabric CAT (Customer Advisory Team). Sure, all the core components of this platform existed in one shape or another before, but the level of integration, ease of use, and new functionality that is unlocked with Fabric is quite amazing.

If you want to hear directly from the leadership and feature PM teams about all the different workloads, you can head over to the Microsoft Fabric Launch Event page to find a list of sessions that were presented on May 24 and 25 showcasing all the functionality that was announced. If you want to just jump right in and watch the full 6+ hours of content, you can catch the on-demand versions for Day 1 and Day 2 over on YouTube.

In a livestream I did earlier this week someone said Fabric was just Synapse repackaged. I can assure you that is not the case. If you have any doubts, just go try the 60-day free trial and see for yourself.

I hope to see you there!

Date: Wednesday July 19, 2023
Time: 6:00 - 8:00 PM Eastern
Where: Keiser University (6430 Southpoint Parkway Suite #100, Jacksonville, FL 32216
Cost: Free!
Registration: Meetup

Introducing Microsoft Fabric: The All-in-One Analytics Solution

The world is awash with data and it is up to data professionals to make sense of it all. Just a few weeks ago Microsoft announced the next generation of analytics at Build: Microsoft Fabric!

Fabric is a complete end-to-end analytics platform bringing together Azure Data Factory, Azure Synapse, and Power BI in a single location and more deeply integrated than ever. It’s built on a SaaS platform which helps deliver innovation more quickly and allowing users to get up and running in seconds. At the core Fabric provides a lake-centric and open data hub using the popular Delta format with built-in security, governance, and compliance throughout. At the edge, Fabric delivers flexibility for data scientists, data warehouse developers, and Power BI users to build and analyze using their unique skill sets.

During this session we will discuss the guiding principles and architecture behind Fabric and show a lot of demos! Whether you’re a data engineer, data scientist, data warehouse developer, or an existing Power BI user, Fabric has something for everyone. You won’t want to miss this session!

I hope to see you there!

Date: Thursday June 15, 2023
Time: 6:00 - 7:30 PM Eastern
Where: Online (Microsoft Teams, register to receive the link)
Cost: Free!
Registration: Meetup

Introducing Microsoft Fabric: The All-in-One Analytics Solution

The world is awash with data and it is up to data professionals to make sense of it all. Just a few weeks ago Microsoft announced the next generation of analytics at Build: Microsoft Fabric!

Fabric is a complete end-to-end analytics platform bringing together Azure Data Factory, Azure Synapse, and Power BI in a single location and more deeply integrated than ever. It’s built on a SaaS platform which helps deliver innovation more quickly and allowing users to get up and running in seconds. At the core Fabric provides a lake-centric and open data hub using the popular Delta format with built-in security, governance, and compliance throughout. At the edge, Fabric delivers flexibility for data scientists, data warehouse developers, and Power BI users to build and analyze using their unique skill sets.

During this session we will discuss the guiding principles and architecture behind Fabric and show a lot of demos! Whether you’re a data engineer, data scientist, data warehouse developer, or an existing Power BI user, Fabric has something for everyone. You won’t want to miss this session!

The default is 1,000 of them and they aren't cheap.

Do you need 1,000? 100? 8,278?

These are all good questions, but let's try to address the "What is a DWU?" question first.

What is a DWU?

A DWU, or Data Warehouse Unit, is the Synapse dedicated SQL pool's blended representation of compute power. Similar to the early days of Azure SQL Database where we only had DTUs, you won't find vCores or memory listed anywhere on the page or documentation.

This number drives a number of things that we will cover in separate posts at a later date, but at the core it combines CPU, memory, and IO. The used DWU percentage will therefore be represented by the maximum of those three metrics. For example, a workload that is very CPU intensive using 60% of the available CPU for the currently selected DWU, but uses very little memory at 10% and a small amount of IO at 15% would show as using 60% of the available DWU. Similarly, an IO intensive workload using 5% CPU, 15% memory, and 40% of the provisioned IO would show as using 40% of the available DWU.

How do I choose the right number of DWUs?

To oversimplify things, if you need less overall performance then go with a lower DWU. If you need more overall performance, then go with a higher DWU.

I'm a former consultant, so I am well trained in saying "it depends". To that end, in reality, the number of required DWUs is very workload specific. Here are a few guiding principals for helping land on the right setting:

• Do you initial development on a very small number of DWUs. Start with 100 DWUs just get the DDL deployed even.

• DW100 to DW500 are all a single compute node with varying levels of compute power. That means you are really running a single node server with the overhead of an MPP engine. Not until DW1000 do you get a second compute node.

• When doing load tests, use at least DW1000. Anything lower shouldn't be considered for production and should only be used for development purposes.

• Use a tool like JMeter to simulate a workload. Be sure to look at the whole workload.

  • Build a simulation for your ETL.

  • Build another for ad hoc user queries.

  • Build another for reporting tool queries.

  • Run them in series or parallel depending on expected loading and query patterns.

  • Use realistic data sizes. Don't build the simulation on 100k records when you're going to have 100 billion records. It doesn't need to be a 100% match on data size, but it should be representative.

• Scale up or down based on the performance observed in your test.

• Don't forget to account for bursts of activity and plan to scale appropriately. A monthly or quarterly load may mean you scale up a few hours in advance or at some logical point where you can take a few minutes of downtime for the scaling operation to complete.

• DWUs drive concurrency. Maximum concurrency, 128 queries, in not unlocked until DW6000c.

• The more CCI rebuilds you want to run in parallel, the more memory you will need, the more DWUs you will need.

• The moral of the story is this...test, test, test.

How do I know what DWU I'm running?

The current SQL pool service level will be shown in the Azure Portal, Synapse Studio, PowerShell, and through the T-SQL query below when connected to the master database.

SQL

SELECT
    d.name AS DatabaseName,
    Edition AS DatabaseEdition,
    service_objective AS ServiceObjective
FROM sys.database_service_objectives AS dso
INNER JOIN sys.databases AS d
    ON dso.database_id = d.database_id

There you have it. Data warehouse units simplified!

It's been a few years since there was an in-person PASS Summit. Some major changes and a couple of years later we are excited to be back in person for what is now the "PASS Data Community Summit".

If SQL Saturday and Data Saturday event sizes are any indication, this conference will be significantly smaller than past years when it was the PASS Summit but with just as much great content and many great networking opportunities. I can't wait to make my way out to Seattle, meet some new people, and present on Synapse + Power BI.

This year I will be presenting a session called "Better Together: Power BI and Azure Synapse Analytics". If you have ever attended a session of mine in the past, you will know I don't like slides. This session will walk through the adventure of creating a Power BI report on Synapse serverless, eventually hitting performance challenges as the dataset gets really big, tuning the report and Synapse layers, and ultimately making the end users happy again.

If you're attending the Data Community Summit and are looking for a fun place to go at 2:30 PM, come on by! I'll also be around on Wednesday at the Microsoft booth and other places around the conference. If you have Synapse questions or just want to meet to talk, come find me!

Ok, I'm back. It is confirmed. I am in fact officially old. I really did write that blog 12 years ago on June 23, 2010. Wow. Anyway...

12 years later that post has received dozens of views from me trying to remember how I did that. Well, I'm here to tell you there is a better way! I didn't discover this because I wanted to improve my code or because I was scanning the release notes for SQL Server to see what new T-SQL functionality has been released in the last few versions. No, I had to find a better way because my previous method which used a variable and the COALESCE function does not work on Azure Synapse Analytics.

The new method: STRING_AGG()

Let's get started with some sample data:

CREATE TABLE dbo.State
    (
        StateID [int],
        StateName [varchar](50)
    )

INSERT INTO dbo.State
SELECT 1, 'Florida'
UNION ALL
SELECT 2, 'Tennessee'
UNION ALL
SELECT 3, 'Georgia'
UNION ALL
SELECT 4, NULL
UNION ALL
SELECT 5, 'Texas'

SELECT * FROM dbo.State

Now, a simple STRING_AGG(expression, separator) and we are good to go!

Quick and simple!

A couple quick notes in closing.

1. Notice that NULL values are ignored and the separator is not added. If you want to include NULL values you'll need to wrap the column name in this example with ISNULL so it would look something like SELECT STRING_AGG(ISNULL(StateName, 'No Name Provided'), ',') FROM dbo.State
2. There is an optional order by operator that can be used to order the list.
3. The data type is determined by the expression. That means if the column is a string it will retain the string properties. MAX fields will result in a MAX data type. Non-max fields will result in the largest possible non-MAX value (VARCHAR(8000) or NVARCHAR(4000)). All other data types result in an NVARCHAR(4000) result.
4. You aren't limited to a literal string like a comma or a pipe for the separator. That's just my most common request. CHAR(13), CHAR(9), and just about any other expression are all valid as well.

May 2022

Read about the May 2022 updates on the Azure Synapse Blog.

• General
  • [See Details] Get connected with the Azure Synapse Influencer program
• SQL
  • [See Details] Data Warehouse Migration guide for Dedicated SQL Pools in Azure Synapse Analytics
  • [See Details] Specify character column lengths... Not anymore!
• Apache Spark for Synapse
  • [See Details] Azure Synapse Dedicated SQL Pool Connector for Apache Spark Now Available in Python
  • [See Details] Manage Azure Synapse Apache Spark configuration
• Synapse Data Explorer
  • [See Details] Synapse Data Explorer live query in Excel
  • [See Details] Use Managed Identities for External SQL Server Tables
  • [See Details] New KQL Learn module (2 out of 3) is live!
  • [See Details] Azure Synapse Data Explorer connector for Microsoft Power Automate, Logic Apps, and Power Apps [Gen...
  • [See Details] Dynamic events routing from event hub to multiple databases
  • [See Details] Configure a database using a KQL inline script as part of JSON ARM deployment template
• Data Integration
  • [See Details] Export pipeline monitoring as a CSV
  • [See Details] Incremental data loading made easy for Synapse and Azure Database for PostgreSQL and MySQL
  • [See Details] User-Defined Functions for Mapping Data Flows [Public Preview]
  • [See Details] Assert Error Handling
  • [See Details] Mapping data flows projection editing
• Synapse Link
  • [See Details] Azure Synapse Link for SQL [Public Preview]

April 2022

Read about the April 2022 updates on the Azure Synapse Blog.

• SQL
  • [See Details] Cross-subscription restore for Azure Synapse SQL [Generally Available]
  • [See Details] Recover SQL pool from dropped server or workspace
• Synapse Database Templates & Database Designer
  • [See Details] Revamped exploration experience
  • [See Details] Clone lake database
  • [See Details] Use wildcards to specify custom folder hierarchies
• Apache Spark for Synapse
  • [See Details] Apache Spark 3.2 [Public Preview]
  • [See Details] Parameterization for Spark job definition
  • [See Details] Notebook snapshot
• Security
  • [See Details] Synapse Monitoring Operator RBAC role [Generally Available]
• Data Integration
  • [See Details] Dataverse connector added to Synapse Data Flows
  • [See Details] Synapse Pipelines Web activity response timeout improvement
• Developer Experience
  • [See Details] Reference unpublished notebooks

March 2022

Read about the March 2022 updates on the Azure Synapse Blog.

• Developer Experience
  • [See Details] Synapse notebooks: Code cells with exception to show standard output
  • [See Details] Synapse notebooks: Partial output is available for running notebook code cells
  • [See Details] Synapse notebooks: Dynamically control your Spark session configuration with pipeline parameters
  • [See Details] Synapse notebooks: Reuse and manage notebook sessions
  • [See Details] Synapse notebooks: Support for Python logging
• SQL
  • [See Details] Column Level Encryption for Azure Synapse dedicated SQL Pools [Generally Available]
  • [See Details] Better performance for CETAS and subsequent SELECTs
• Apache Spark for Synapse
  • [See Details] Synapse Spark Common Data Model (CDM) Connector [Generally Available]
  • [[](https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/azure-synapse-analytics-march-update-2022/ba-p/3269194#TOCREF_10)See Details] Spark Dedicated SQL Pool (DW) Connector: Performance Improvements
  • [See Details] Synapse Spark Dedicated SQL Pool (DW) Connector: Support for all Spark Dataframe SaveMode choices (...
  • [See Details] Accelerate Spark execution speed using the new Intelligent Cache feature [Public Preview]
• Security
  • [See Details] Azure Synapse Analytics now supports Azure Active Directory (Azure AD) only authentication
  • [See Details] API support to raise or lower Workspace Managed SQL Server Dedicated SQL Minimal TLS version
• Data Integration
  • [See Details] Flowlets and CDC Connectors [Generally Available]
  • [See Details] sFTP connector for Synapse data flows
  • [See Details] Data flow improvements to Data Preview
  • [See Details] Pipeline script activity

February 2022

Read about the February 2022 updates on the Azure Synapse Blog.

• SQL
  • [See Details] More consistent query execution times for Serverless SQL Pools
  • [See Details] The OPENJSON function makes it easy to get array element indexes
• Data Integration
  • [See Details] Upsert supported by Copy Activity
  • [See Details] Transform Dynamics Data Visually in Synapse Data Flows
  • [See Details] Connect to your SQL sources in data flows using Always Encrypted
  • [See Details] Capture descriptions from Asserts in Data Flows
  • [See Details] Easily define schemas for complex type fields

January 2022

Read about the January 2022 updates on the Azure Synapse Blog.

• Apache Spark for Synapse
  • [See Details] 4 New database templates [Public Preview]
• Machine Learning
  • [See Details] Improvements to the SynapseML library
• Security
  • [See Details] Azure Synapse Analytics security overview
  • [See Details] TLS 1.2 required for new workspaces
• Data Integration
  • [See Details] Data quality validation rules using Assert transformation
  • [See Details] Native data flow connector for Dynamics
  • [See Details] IntelliSense and auto-complete added to pipeline expressions
• Synapse SQL
  • [See Details] COPY schema discovery for complex data ingestion
  • [See Details] HASHBYTES easily generates hashes in Serverless SQL

December 2021

Read about the December 2021 updates on the Azure Synapse Blog.

• [See Details] Finding Synapse Monthly Update blogs
• Apache Spark in Synapse
  • [See Details] Additional notebook export formats
  • [See Details] Three new chart types in notebooks
  • [See Details] Reconnect notebooks to Spark sessions
• Data Integration
  • [See Details] Map Data tool [Public Preview], a no-code guided ETL experience
  • [See Details] Quick Reuse of Spark clusters
  • [See Details] External call transformation
  • [See Details] Flowlets [Public Preview]

This post isn’t about Azure Monitor though. Not directly anyway.

Azure Synapse is no different than any other application or service in your environment. It serves a purpose in the architecture and as a result needs special attention from people with a specific skillset, in this case DBAs and developers. That’s where Azure Monitor comes into the picture and what we will be discussing today: how to make sense of the integration between auditing in Azure Synapse and Azure Monitor. The focus will be on Azure Synapse Analytics Dedicated SQL Pools (Gen 2) specifically, but there is integration for monitoring serverless SQL pools, Spark pools, pipelines, and general workspace operations.

First, those with an eye for detail will notice that this post has Log Analytics in the title and not Azure Monitor. That’s true. A very brief explanation so everyone has things straight in their heads. I put Log Analytics in the title because when you set up auditing or diagnostics the option is labeled “Send to Log Analytics workspace” not “Send to Azure Monitor” and I wanted people to be able to find this post. Azure Monitor is a set of functionalities for collecting and analyzing logs. There are some prebuilt integrations and visualizations with some Azure services like Key Vault or Storage Accounts Then there is Log Analytics, a functionality inside Azure Monitor which also happens to be where Synapse writes its logs, for storing and querying all log data. Queries are written in a Kusto Query Language or KQL.

Second, let’s be sure a few very important details are clear. When referring to Azure Synapse there are two distinct but very similar services. They are both the same Microsoft cloud MPP database engine, both cover the same general T-SQL surface area, both scale in terms of DWUs, both talk about “Azure Synapse Analytics” in the documentation and description, but one is deployed to an Azure SQL Server, and one is deployed to a Workspace.

We need to be very clear about this because they both operate slightly different. Again, this post is focused on Azure Synapse Analytics Dedicated SQL Pools (Gen 2). An alternate post is available for Dedicate SQL Pools (Formerly SQL DW) if you need information about that deployment option.

Let’s get started.

What are my options?

There are two types of logging that can be enabled: Auditing and Diagnostics. Auditing tracks a set of database events like queries and login attempts. Diagnostics will make copies of a specific set of system DMVs periodically which can be helpful because Synapse only stores a few thousand records depending on the DMV.

Everything from here assumes that you have a Log Analytics workspace created and a Synapse Workspace with a Gen 2 dedicated SQL pool deployed.

Enabling Auditing

Auditing can be enabled at the workspace level, which will cover all databases on the workspace automatically, or at the individual database level.

First, navigate to your Synapse Analytics Workspace or dedicated SQL pool in the Azure Portal. My screenshots will show the configuration from the dedicated SQL pool, but the same setting can be found under the label of Azure SQL Auditing at the workspace level. From here, select Auditing from the Security section.

Next, toggle the Enable Azure SQL Auditing to the on position. Next, check the boxes for the locations where you would like the log to be written, in this example we are going to focus on Log Analytics. Select a log analytics workspace to which the data will be written. Click Save once complete.

Enabling Diagnostics

While Auditing can be turned on for a database by either enabling it at the workspace or database level, diagnostics must be enabled at the database level. There are diagnostics at the workspace level, but those are different metrics and do not apply to monitoring the dedicated SQL pool.

If you’re not there already, navigate in the Azure Portal to your dedicated SQL pool. Select Diagnostic Settings from the Monitoring section.

Next, click the Add diagnostic setting button.

Provide a name for this diagnostic setting. Check the box next to Send to Log Analytics workspace. Select the subscription and workspace to which the DMVs will be written. Select which DMVs you want to log. Finally, click Save.

Remember, the diagnostic settings each correspond to a different DMV that will be copied to your Log Analytics workspace. If you need additional DMVs you’ll need to roll your own auditing solution at this time.

Reading the Logs

Data will take several minutes to show up in Log Analytics. I’ve had the initial population take up to 30 minutes. After that the data should show up much more quickly, within several minutes. Connect to your database and run a query or two to ensure we have some user queries show up in the logs.

For my example I have run the following query:
SELECT COUNT(*), GETDATE(), DB_NAME() FROM IMDB.stage_Person

To view the logs, you can navigate to your log analytics workspace, or select Logs from your dedicated SQL pool’s Monitoring section. This can be found directly beneath the Diagnostic Settings option used in the last section.

Viewing the Audit Log

Within the Log Analytics Workspace you will see a group of tables with the prefix Synapse* and at least one table prefixed with SQL*. More tables may appear if you are using auditing for Azure SQL Database. The audit setup in the first section of this post will be logged to the SQLSecurityAuditEvents table.

This sample query pulls a few of the relevant columns from the SQLSecurityAuditEvents table.

SQLSecurityAuditEvents
| project
    ResourceGroup,
    LogicalServerName,
    DatabaseName,
    EventTime,
    Category,
    ActionName,
    IsServerLevelAudit,
    Succeeded,
    tostring(SessionId),
    ClientIp,
    HostName,
    ServerPrincipalName,
    DurationMs,
    Statement,
    TenantId,
    _ResourceId

Here you can see the query I ran in the log with an action of BATCH COMPLETED.

Viewing the Diagnostic Logs (DMVs)

Where all audit events are logged to the same table, regardless of the event type, each of the DMVs enabled in Diagnostic settings gets logged to its own table. Expanding on the list of DMVs from earlier in the post we can see the Log Analytics table closely matches the diagnostic setting name with a prefix of SynapseSqlPool.

This sample query pulls a few of the relevant columns from the SynapseSqlPoolExecRequests table.

SynapseSqlPoolExecRequests
| summarize
    TimeGenerated = max(TimeGenerated),
    ResourceGroup = any(ResourceGroup),
    LogicalServerName = any(LogicalServerName),
    DatabaseId = any(toreal(DatabaseId)),
    StartTime = max(StartTime),
    EndCompileTime = max(EndCompileTime),
    Category = any(Category),
    Status = min(Status),
    Command = any(Command)
    by RequestId

Here you can see the query I ran in the SynapseSqlPoolExecRequests table.

Wrapping it up

There is a lot more that can be accomplished with KQL queries in Log Analytics. You may even take the queries found in the accompanying Dedicate SQL Pools (Formerly SQL DW) post and combine them to give a single view of all your dedicate SQL pools.

The key take learning from this walkthrough is:

1. Auditing and Diagnostics are different options and enabled in different locations.
2. Auditing writes all the data to a single table no matter the action.
3. Diagnostics pull a few key DMVs from Synapse and write them to the log. Each DMV is written to its own table in Log Analytics.
4. Something not discussed is that there is the potential on highly used systems for some audit or diagnostic records could be missed.

This post isn’t about Azure Monitor though. Not directly anyway.

Azure Synapse is no different than any other application or service in your environment. It serves a purpose in the architecture and as a result needs special attention from people with a specific skillset, in this case DBAs and developers. That’s where Azure Monitor comes into the picture and what we will be discussing today: how to make sense of the integration between auditing in Dedicated SQL Pools (Formerly SQL DW) and Azure Monitor. The focus will be on Dedicated SQL Pools (Formerly SQL DW) not Azure Synapse Analytics, but more on that in a moment.

First, those with an eye for detail will notice that this post has Log Analytics in the title and not Azure Monitor. That’s true. A very brief explanation so everyone has things straight in their heads. I put Log Analytics in the title because when you set up auditing or diagnostics the option is labeled “Send to Log Analytics workspace” not “Send to Azure Monitor” and I wanted people to be able to find this post. Azure Monitor is a set of functionalities for collecting and analyzing logs. There are some prebuilt integrations and visualizations with some Azure services like Key Vault or Storage Accounts Then there is Log Analytics, a functionality inside Azure Monitor which also happens to be where Synapse writes its logs, for storing and querying all log data. Queries are written in a Kusto Query Language or KQL.

Second, let’s be sure a few very important details are clear. When referring to Azure Synapse there are two distinct but very similar services. They are both the same Microsoft cloud MPP database engine, both cover the same general T-SQL surface area, both scale in terms of DWUs, both talk about “Azure Synapse Analytics” in the documentation and description, but one is deployed to an Azure SQL Server, and one is deployed to a Workspace.

We need to be very clear about this because they both operate slightly different. Again, this post is focused on Dedicated SQL Pools (Formerly SQL DW). An alternate post is available for Azure Synapse Analytics Dedicated SQL Pools (Gen 2) if you need information about that deployment option.

Let’s get started.

What are my options?

There are two types of logging that can be enabled: Auditing and Diagnostics. Auditing tracks a set of database events like queries and login attempts. Diagnostics will make copies of a specific set of system DMVs periodically which can be helpful because Synapse only stores a few thousand records depending on the DMV.

Everything from here assumes that you have a Log Analytics workspace created and a Dedicated SQL Pools (Formerly SQL DW) deployed.

Enabling Auditing

Auditing can be enabled at the logical server level, which will cover all databases on the logical server automatically, or at the individual database level.

First, navigate to your logical SQL server or dedicated SQL pool in the Azure Portal. My screenshots will show the configuration from the dedicated SQL pool, but the same setting can be found under the label of Auditing at the logical SQL server level. From here, select Auditing from the Security section.

Next, toggle the Enable Azure SQL Auditing to the on position. Next, check the boxes for the locations where you would like the log to be written, in this example we are going to focus on Log Analytics. Select a log analytics workspace to which the data will be written. Click Save once complete.

Enabling Diagnostics

While Auditing can be turned on for a database by either enabling it at the logical SQL server or database level, diagnostics must be enabled at the database level. There are no diagnostics at the logical SQL server level.

If you’re not there already, navigate in the Azure Portal to your dedicated SQL pool. Select Diagnostic Settings from the Monitoring section.

Next, click the Add diagnostic setting button.

Provide a name for this diagnostic setting. Check the box next to Send to Log Analytics workspace. Select the subscription and workspace to which the DMVs will be written. Select which DMVs you want to log. There is an additional SQLSecurityAuditEvents entry in my screenshot which is there because I already enabled the database audit. We'll ignore that for the time being. Finally, click Save.

Remember, the diagnostic settings each correspond to a different DMV that will be copied to your Log Analytics workspace. If you need additional DMVs you’ll need to roll your own auditing solution at this time.

Reading the Logs

Data will take several minutes to show up in Log Analytics. I’ve had the initial population take up to 30 minutes. After that the data should show up much more quickly, within several minutes. Connect to your database and run a query or two to ensure we have some user queries show up in the logs.

For my example I have run the following query:
SELECT *, GETDATE(), DB_NAME() FROM dbo.MyTable

To view the logs, you can navigate to your log analytics workspace, or select Logs from your dedicated SQL pool’s Monitoring section. This can be found directly beneath the Diagnostic Settings option used in the last section.

Viewing the Audit Log

Within the Log Analytics Workspace you may see a group of tables with the prefix Synapse*. These tables correspond to diagnostic settings configured on the Synapse Analytics Workspaces. If you do not have any Synapse Workspaces these tables will likely not show up. More tables may appear if you are using auditing for Azure SQL Database. The audit setup in the first section of this post will be logged to the AzureDiagnostics table.

This sample query pulls a few of the relevant columns from the AzureDiagnostics table by narrowing down the data to the SQLSecurityAuditEvents category which is used for the Audit log. For Synapse Workspace Dedicated SQL Pools, as opposed to the "formerly SQL DW" SQL Pools that are the focus of this post, the data is written to a separate table called SQLSecurityAuditEvents rather than being a category of log within the AzureDiagnostics table.

AzureDiagnostics
| where Category == "SQLSecurityAuditEvents"
| project
        ResourceGroup,
        LogicalServerName_s,
        database_name_s,
        event_time_t,
        Category,
        action_name_s,
        tobool(is_server_level_audit_s),
        tobool(succeeded_s),
        tostring(session_id_d),
        client_ip_s,
        host_name_s,
        server_principal_name_s,
        tolong(duration_milliseconds_d),
        statement_s,
        TenantId,
        _ResourceId
| project-rename
        ResourceGroup,
        LogicalServerName=LogicalServerName_s,
        DatabaseName=database_name_s,
        EventTime=event_time_t,
        Category,
        ActionName=action_name_s,
        IsServerLevelAudit=is_server_level_audit_s,
        Succeeded=succeeded_s,
        SessionId=session_id_d,
        ClientIp=client_ip_s,
        HostName=host_name_s,
        ServerPrincipalName=server_principal_name_s,
        DurationMs=duration_milliseconds_d,
        Statement=statement_s,
        TenantId,
        _ResourceId

Here you can see the query I ran in the log with an action of BATCH COMPLETED.

Viewing the Diagnostic Logs (DMVs)

Where Synapse Workspaces write DMV data to separate tables, for Dedicated SQL Pools (Formerly SQL DW) each of the DMVs enabled in Diagnostic settings gets logged to the same AzureDiagnostics table used in the Auditing section above, but with a category that corresponds to the diagnostic setting/System DMV.

This sample query pulls a few of the relevant columns from the AzureDiagnostics table by narrowing the data down to the ExecRequests Category. This data corresponds to the system DMV sys.dm_pdw_exec_requests.

AzureDiagnostics
| where Category == "ExecRequests"
| project
    ResourceGroup,
    LogicalServerName_s,
    DatabaseId_d,
    StartTime_t,
    EndCompileTime_t,
    Category,
    Status_s,
    RequestId_s,
    Command_s
| summarize
    ResourceGroup = any(ResourceGroup),
    LogicalServerName = any(LogicalServerName_s),
    DatabaseId = any(DatabaseId_d),
    StartTime = max(StartTime_t),
    EndCompileTime = max(EndCompileTime_t),
    Category = any(Category),
    Status = min(Status_s),
    Command = any(Command_s),
    record_count = count()
    by RequestId_s

Here you can see the query I ran focused on the ExecRequests category.

Wrapping it up

There is a lot more that can be accomplished with KQL queries in Log Analytics. You may even take the queries found in the accompanying Azure Synapse Analytics Dedicated SQL Pools (Gen 2) post and combine them to give a single view of all your dedicate SQL pools.

The key take learning from this walkthrough is:

1. Auditing and Diagnostics are different options and enabled in different locations.
2. Auditing writes all the data to a single table no matter the action.
3. Diagnostics pull a few key DMVs from Synapse and write them to the log. Each DMV is written to the same table in Log Analytics each with a different category.
4. Something not discussed is that there is the potential on highly used systems for some audit or diagnostic records could be missed.

Maybe my issue isn't with Key Vault so much as it is with soft-delete.

While doing some testing for TDE protectors on Managed Instance I had to remove a key and recreate it for part of my test setup. The key I deleted earlier in the day was named ManagedInstance. So when I went to create a new key called ManagedInstance in the Azure Portal I received an error message. No big deal, I had this one in the bag. Go to PowerShell, find the key, remove the key, go back to the portal and try again. Unfortunately, I hit a snag. Let's walk through the code and see where I went wrong.

To remove a soft-deleted key you must first locate the key's name or Id. To do this we will use the Get-AzKeyVaultKey cmdlet with the parameter -InRemovedState to show all the deleted, but not really deleted (aka soft-deleted) keys.

Get-AzKeyVaultKey -VaultName "bschacht-kv" -InRemovedState

Make note of the name or Id and move on to the next cmdlet and we will be done...or we should have been done anyway. We will use Remove-AzKeyVaultKey, specify the key name to be removed, make note that this key is in a removed state and run the cmdlet. Except here is where I ran into the somewhat unhelpful error message Remove-AzKeyVaultKey : Operation returned an invalid status code 'Forbidden'.

Remove-AzKeyVaultKey -VaultName "bschacht-kv" -Name "ManagedInstance" -InRemovedState

It turns out this error message stems from the user running the command (myself in this case) doesn't not have purge permissions explicitly granted in the Key Vault. Running this cmdlet requires the purge permission, no inherited RBAC or explicitly granted RBAC roles no matter how high up the food chain (Contributor, Owner, etc.) will give you the ability to remove a soft-deleted key. You must go to the Azure Portal, navigate to your Key Vault, go to Access Policies, and grant the user running this cmdlet the Purge permission for keys which is in a special section on the dropdown menu called "Privileged Key Operations".

Now, let's run the Remove-AzKeyVaultKey cmdlet again, verify that we are sure we want to remove the key permanently, and no response means success!

There you have it. In the end it was a very simple fix. Unfortunately that error message seems to be less than well documented. Thankfully, if you go look at the documentation for the cmdlet under example 3 (which, let's be honest, who does that unless they run into a problem and have to go looking for an answer) you will in fact see the purge permission noted. Too bad I don't read instructions. I just go run things and see what happens. It's a great way to learn...as long as you aren't doing it in production.

A few prerequisites for today:

• Azure SQL Managed Instance (any service tier, any size)
• A database (any user database will do, even an empty one)
• Azure Key Vault (with soft-delete enabled)
• Storage Account (optional, if you want to test the backup/restore process)

Switching to Bring Your Own Key

Bringing your own key for TDE is actually not a requirement to use Managed Instance. By default, the service will manage its own TDE key. However, that comes with a few limitations. The main one being you cannot create user initiated backups when using the service-managed key. The built-in backup/restore functionality still works just fine with service-managed keys though.

The key settings can be found by navigating the Azure Portal to your Managed Instance and clicking on the Transparent Data Encryption option in the service navigation panel. Then change the

The next step is to identify the key that will be used for encrypting the databases. This can be done in one of two ways: choosing a Key Vault and Key from inside the tenant or specifying a key identifier URL for a particular key. The second option is particularly useful if the user doing the configuration does not have access to the list of Key Vaults/Keys but is given a key identifier that should be used. The two configuration options can be seen in the following screenshot.

Important Notes

A few important pieces of information to note on from these settings.

First, the Managed Instance must be granted access through a Key Vault access policy. If that has not already been completed an attempt will be made to add the Managed Instance with the appropriate permissions (Get, Wrap Key, Unwrap Key). The access policy is defined and enforced at the vault level, not the individual key, secret, or certificate level. So an instance with Get access to one key in this vault will have Get access to all the keys in this vault.

The second piece of important information is the check box labeled Make the selected key the default TDE protector. By checking that box, all databases will be encrypted using the selected key. The situation may arise where you would like to simply restore a database protected with a particular key, but you do not wish to encrypt the databases on the server with that key. In that scenario, you will want to add the key to the Managed Instance, but NOT select the option to make it the default TDE protector.

Next, the Managed Instance and Key Vault must be in the same Active Directory tenant. At this time it is not supported to backup keys from one subscription and move them to another subscription, the keys must remain in the same subscription but can be moved to another vault someplace else in the same geography (moving between Key Vaults in North America that reside in the same subscription is supported, moving between a Key Vault in North America and one in Europe is not supported even if they are in the same subscription.

Finally, changing the default TDE protector does not encrypt existing backups with the new key. So it is important for business continuity that you retain the prior TDE key(s) in order to be able to restore backups. If you switch from customer-managed key back to service-managed key you must retain the copy of your key in Key Vault at least until the system's point-in-time restore window has rolled off for the customer-managed keys (i.e. retain the customer-managed key for 7-35 days depending on your configuration after switching back to service-managed key).

There is one and only one key that will protect the databases on an instance.

• Key marked as "Make the selected key the default TDE protector."
  • Can restore databases encrypted with this key.
  • Will be the key used to encrypt the user databases on the instance.
  • A database restored with this key will also be encrypted with this key.
• Key NOT marked as "Make the selected key the default TDE protector."
  • Can restored databases encrypted with this key.
  • A database restored with this key will be encrypted with the key marked as the protector after the restore process is completed.

See it in Action - Changing to Customer-Managed Key

As a starting point I have a Managed Instance with a single database, TDETesting, that is encrypted with the service-managed key.

We can use a simple T-SQL query to get the encryptor thumbprint. This same query will help us identify when the key has changed.

SELECT
    DB_NAME(database_id) AS database_name,
    encryption_state,
    CASE encryption_state
        WHEN 0 THEN 'No Encryption Key Present. Database Not Encrypted.'
        WHEN 1 THEN 'Database Unencrypted'
        WHEN 2 THEN 'Database Encryption in Progress'
        WHEN 3 THEN 'Database Encrypted'
        WHEN 4 THEN 'Encryption Key Change in Progress'
        WHEN 5 THEN 'Database Decryption in Progress'
        WHEN 6 THEN 'Certificate or Key Change in Progress'
        ELSE 'Unknown Status'
        END AS encryption_state_descriptoin,
    encryptor_thumbprint
FROM sys.dm_database_encryption_keys

We can see that based on the query results the current thumbprint is 0x051082BA88D2882F551C530B74EB6F4380843029.

Before the switching to the customer-managed key, this is what the access policy settings look like on my Key Vault. The only user that has rights to do anything is myself.

From my Managed Instance I will switch to customer-managed key.

Select my Key Vault.

And on the select a key blade, I will choose create a new key as I do not already have a key to use.

I will keep the option on generate to create a key and provide a name. I'm going to use ManagedInstance, but there is not a specific name that is required. Then Click create.

With all the options configured click Save.

After the settings are applied we can see the change in access policy on the Key Vault has added my Managed Instance with Get, Wrap, and Unwrap permissions.

Additionally, running our T-SQL script we can see the encryptor thumbprint has now changed from 0x051082BA88D2882F551C530B74EB6F4380843029 to 0x71ABFFF1EAA10687BD43878C75E7F2D1744E285C.

With the customer-managed key in place, we can now run a create a user backup of the database.

BACKUP DATABASE TDETesting TO URL = 'https://bschachtstor.blob.core.windows.net/backup/TDETesting.bak' WITH COPY_ONLY

If we then look at the files in that backup, we will see that it is encrypted with the same thumbprint as our database showed after switching to customer-managed keys.

RESTORE FILELISTONLY FROM URL = 'https://bschachtstor.blob.core.windows.net/backup/TDETesting.bak'

See it in Action - Restoring a Backup

What happens when we need to restore a database that has been backed up under a different key? In this particular case I have a backup that was created under a different customer-managed key. This screenshot shows a different thumbprint than the system or current customer-managed key, 0x190256228610BBE409C0345597D49ABB5A40EFA6.

System key thumbprint: 0x051082BA88D2882F551C530B74EB6F4380843029
Current key thumbprint: 0x71ABFFF1EAA10687BD43878C75E7F2D1744E285C
Backup key thumbprint: 0x190256228610BBE409C0345597D49ABB5A40EFA6

In the Azure Portal, I changed the key to the new ManagedInstance-AlternateKey that I added to my Key Vault. However, I did not choose the Make the selected key the default TDE protector option as I want to continue to encrypt my databases with the key used previously in this example but restore databases that are encrypted with the ManagedInstance-AlternateKey.

After saving the change I can now run the restore of my database that is encrypted with the ManagedInstance-AlternateKey (thumbprint 0x190256228610BBE409C0345597D49ABB5A40EFA6). After the restore completes we can see that the database encryptor has been switched to the thumbprint 0x71ABFFF1EAA10687BD43878C75E7F2D1744E285C that is used on my existing database. As mentioned previously, the current implementation is such that all databases are encrypted with the same key. So while the key ending in EFA6 is required for the restore process the database is then encrypted with the TDE protector key ending in 285C.

What Keys Are Associated with my Managed Instance?

We have seen how we can assign a customer-managed key to Managed Instance, add an additional key for restore purposes, how to see which key is encrypting the database, and how to restore backups encrypted with non-TDE Protector keys. There can be confusion as to what keys are available though because the Azure Portal only shows the last key assigned. In our demo walkthrough that wasn't even the TDE Protector key. How then do we find what keys are available to our managed instance for restore commands and what key is the current TDE Protector?

For that we go to PowerShell.

Get-AzSqlInstanceKeyVaultKey will give a list of all keys currently associated with your managed instance. In order for a backup file to be restored to your Managed Instance the key with which the backup was encrypted must be listed. In my example there is a service-managed key and two customer-managed keys with the thumbprints from earlier in this post.

Get-AzSqlInstanceKeyVaultKey -ResourceGroupName "ManagedInstance-RG" -InstanceName "bschacht-sqlmi01"

However, there is no indication of the current TDE Protector. For that we will go to a different PowerShell cmdlet, Get-AzSqlInstanceTransparentDataEncryptionProtector.

Get-AzSqlInstanceTransparentDataEncryptionProtector -ResourceGroupName "ManagedInstance-RG" -InstanceName "bschacht-sqlmi01"

Removing a key is also a simple operation with the cmdlet Remove-AzSqlInstanceKeyVaultKey.

Remove-AzSqlInstanceKeyVaultKey -ResourceGroupName "ManagedInstance-RG" -InstanceName "bschacht-sqlmi01" -KeyId https://bschacht-kv.vault.azure.net/keys/ManagedInstance-AlternateKey/2603353efae04fbab840385dd43a590b

Wrapping Up

What other questions do you have about using TDE with Azure SQL Database Managed Instance? Let me know in the comments and I'll see if I can work through some examples with you.

As I work through this process I am going to post the scripts that I use. These are by no means meant to take back and use to build your enterprise environment. There are many not so best practices in the scripts, lots of copy and paste where there should be variables, no logic to check if resources exist or remove if they do exist, and a bunch of things that could likely be done more efficiently.

Getting Started

All you need to get started is a PowerShell window. I'm using Visual Studio Code but this could just as easily been done with the PowerShell ISE. Visual Studio Code is a great cross platform tool that can be downloaded here.

Once your favorite code editor is open it's time to install the Az module which is used for interacting with Azure. If you've used PowerShell for Azure in the past you may remember the AzureRM (Azure Resource Manager) module. This is the next evolution of the AzureRM module which has been deprecated. Make sure you are running your application as an administrator and run Install-Module Az.

Install-Module Az

The Az module cmdlets follow a pretty general format.

• Get-Something will pull back all of that particular resource. For instance, Get-ResourceGroup will list all the resource groups. Get-AzVirtualNetwork will return all the virtual networks. It's an easy way to validate what you have just created or modified. Most of these commands are going to require you to specify at least a resource group.
• New-Something will create a new resource. For instance, New-AzVirtualNetwork will create a new virtual network. Almost all of these commands require a name for the resource, the location to which the resource will be deployed (Azure region), and the resource group in which to create the resource.
• Remove-Something will remove the specified resource. Again, most of these commands will require at least a name and a resource group to be specified.
• Set-Something will apply a change to a resource. For instance, Set-AzNetworkSecurityGroup will apply a change made to an NSG object. The change isn't actually committed and reflected in the resource until the Set command is run.

Next we will connect to our Azure account and tell our session which subscription (assuming there is more than one available) to use. We will log in, list the subscriptions, the current context, and if necessary, change subscriptions.

#Connect to Azure account.
Login-AzAccount
#List all the subscriptions and see the current subscription context.
Get-AzSubscription
Get-AzContext
#Set the context to the appropriate subscription.
Select-AzSubscription -SubscriptionId "########-####-####-####-#########" | Set-AzContext

Let's Break Some Stuff

What kind of learning exercise would this be if we didn't try to break some things. Just for the heck of it, let's unregister all of the providers. Providers tell us what resources we are able to spin up inside a given subscription. It's basically like a master switch at the subscription level for enabling and disabling different Azure features. For instance, if you don't register the Microsoft.DataFactory resource provider then you won't be able to create a new Azure Data Factory. The options will still show up in the portal, it will just give an error if you try to create one.

#Just because I like to break things to see what happens. Let's unregister all the resource providers.
Get-AzResourceProvider | Unregister-AzResourceProvider
#Interestingly enough, there were a handful that I was unable to disable.
Get-AzResourceProvider | Format-Table

Fortunately and unfortunately this didn't quite break things. I was really hoping that I was going to start getting error messages when attempting to add a storage account that said I needed to go register the provider. What happens, in reality, is that if you have the permissions to enable a particular provider it will automatically register when a new provider is needed. For my little test this meant that adding new resources automatically registered the necessary providers because I'm an Owner (aka administrator) on my subscription. Test somewhat failed but still an interesting experiment. I went and looked what providers were registered after a few later commands, but in practice I kept writing code, running it, and forgetting to go check the provider list.

Resource Groups

A vital starting point for the remainder of this series is creation of Resource Groups. These are simply logical containers where you can set restrictions if you so choose. Limit the resources that can be used, the location where they can be deployed, and set permissions that will cascade to the resources therein. We will start by setting up a few different resource groups that will be used in the future.

#Create a few core resource groups.
New-AzResourceGroup -Name "ManagedInstance-RG" -Location "eastus2"
New-AzResourceGroup -Name "PlatformServices-RG" -Location "eastus2"
New-AzResourceGroup -Name "SqlDatabase-RG" -Location "eastus2"
New-AzResourceGroup -Name "Storage-RG" -Location "eastus2"
New-AzResourceGroup -Name "VirtualMachine-RG" -Location "eastus2"
New-AzResourceGroup -Name "VirtualNetwork-RG" -Location "eastus2"

What location should I use?

Location. Location. Location. It's said of real estate all the time and the Azure cloud is no different. Every single resource that is created will ask you for a location which ties back to an Azure region. In general, you'll want to choose the closest region and use the same region for all your resources. The challenge, however, is that not all services are available in all regions. Certain VM sizes are only in certain places. Services light up in some regions ahead of others depending on hardware availability. There will be times when everything can't sit in the same region so you'll want to choose another one that's close by as a secondary or one across the country that will be used for DR or bringing the solution closer to a geographically distributed usergroup.

How do you know what location options there are? How do you know what the right name is to use for the location East US 2, or West Europe? As with all things...run a PowerShell command.

Get-AzLocation

This command will provide a list of regions and the services available in each region. The location attribute is the value that should be placed into other cmdlets when prompted for the "location" parameter.

Wrapping Up

At this point our shell is ready. Our code editor is configured. We've installed the necessary module to interact with Azure. We've connected to our subscription and created resource groups to hold everything that will be built in the coming posts.

I would encourage you to have the Azure Portal open at the same time you are running the PowerShell commands. Run a command, try to pull back the changes in PowerShell, and also go check the portal to see what has been created. Does it match up with what you expected to see? If not, do some research and try to figure out why and if there is anything you can do to control the created resources. Maybe that's a disk automatically created with a name you don't like, a virtual cluster, a resource group, etc. This is the time to learn it so when you're creating scripts to deploy to production you know exactly what the code is going to do.

Fortunately and unfortunately we live in a time when technology is constantly changing and that means we need tools that are able to adapt and keep up with that change. I've used Visual Studio Code for a while just to edit scripts but never really for my day to day code development. I figured that if I'm going to be rebuilding my Azure sandbox, I may as well do it all in PowerShell. Since the PowerShell ISE is no longer being updated I also figured that I should make the move over to Visual Studio Code for as much of my work as I can. Just this week, Microsoft announced that Visual Studio was going online so this is a perfect time to try and force myself into the VS Code world.

So here I go. Into VS Code. I will try my best to change with the times and move over to new software and new tools that promise a great big beautiful tomorrow. Maybe this will finally get me to use Azure Data Studio full time (or close to full time) too considering Data Studio is built on top of VS Code.

I'll try to keep this post up to date with the things I'm doing along the way over these couple weeks while I do my sandbox rebuild. It will also be my reference in the future for "what the junk is happening and why can't I do what I want to do?!".

Thoughts with Brad

I'm in a whole new world. I feel a bit lost right now. Things look pretty. I know this will do everything I want it to do but nothing does quite what I want it to do out of the box. I've gotten far to used to SQL Server Management Studio. Here we go.

I'm noticing many of the shortcuts that I'm used to are there but are just a little different. For instance, Alt + Shift + Up/Down Arrow is now Ctrl + Alt + Shift + Up/Down Arrow.

I may be starting to get the hang of this. I've got a couple extensions. I've got some keyboard shortcuts. I've customized some settings. I still don't feel quite as at home as I do in other tools. For some reason it feels like VS Code isn't quite as polished and as nice of a UI or user experience as other tools. I guess that's what happens when you use a single tool to rule them all though.

I decided to hide the minimap. I don't know that it's super useful for me, but I am sure some will love it. Added that to the settings file.

I'm considering trying out the dark mode for a few days. I generally have my settings for Windows, Visual Studio, Management Studio, on the Light themes that are now shipping. A LOT of people are dark mode fans though, so maybe I'll try it and see if I can get used to it. In the past I haven't really given it a fair shot. We shall see. Not right now though.

Hmmm. This has already come in handy. If you highlight some text then press either " (quote) or ' (apostrophe) on your keyboard it will wrap everything that is highlighted in single or double quotes. Nice little time saver. Nifty.

Extensions

Extension are proving to be quite useful! I knew I was going to need things like the SQL tooling and PowerShell, but there are a lot of neat extensions out there. I'm hoping that everything just works when I start to do this all in Visual Studio online.

PowerShell
Allows me to develop PowerShell scripts right inside VS Code. Adds PowerShell language support and allows me to debug code right inside VS Code too. I may have to check out PowerShell Pro Tools at some point in the future too as I do miss the module explorer from the PowerShell ISE where it was searchable, showed parameters, and had descriptions. I want to see if the Pro Tools one is any better than the one in the basic PowerShell extension.

Theme by Language
Changes the color theme based on the language selected in a given editor tab. This is really useful for making PowerShell look like PowerShell but have the other tabs use the default theme. This helps me feel at home when I'm writing code across multiple languages at the same time. Update: A day or so later I actually got rid of this extension. I think it's an awesome extension and works quite well and if you want to have a more familiar interface by all means use it. I'm trying to go as stock as possible to start with though, and I want to be able to see what my customers are going to see when they fire up the tool and use it. I'll definitely remember this one for the future and will likely add it back in not too far down the road.

Zoom Bar
It's the small things in life that are important. I like having a little icon on the status bar at the bottom of the screen to reset the zoom or select a custom zoom level. This is a minor one that I added because of unfamiliar behavior with the scroll to zoom feature. Not sure if it will stay with me long term or not. The one time I used it may be the only time. I'll reevaluate in a week or so. Update: I still have the extension enabled, but I haven't really gotten any use out of it since I found the setting to enable Ctrl + scroll to change the zoom.

SQL Server (mssql)
I'm to the point in my development environment configuration that I need to start doing some SQL work. Where else do I turn but the SQL Server extension. This experience will follow me in some ways over to Azure Data Studio, but for the moment I'm going to stick around in VS Code and kick the tires on this extension to see where it takes me.

Settings

All the user defined settings are stored in a file called settings.json that you can pull up and easily edit. Alternatively, you can edit the settings through the GUI. This can be edited by pressing Ctrl + Shift + P then typing in Preferences: Open Settings (JSON). If it doesn't come up then you'll need to navigate to the folder where that is stored and open the file.

C:\Users\UserNameHere\AppData\Roaming\Code\User\settings.json

{
    "editor.mouseWheelZoom": true,
    "workbench.colorTheme": "Default Light+",
    "powershell.integratedConsole.focusConsoleOnExecute": false,
    "terminal.integrated.shell.windows": "C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
    "window.menuBarVisibility": "default",
    "workbench.statusBar.visible": true,
    "workbench.activityBar.visible": true,
    "editor.minimap.enabled": false,
    "window.zoomLevel": 0
}

Shortcut Keys

Keyboard shortcuts are stored in a file similar to the settings.

C:\Users\UserNameHere\AppData\Roaming\Code\User\keybindings.json

Ctrl + Shift + P: Show all command

Ctrl + P: Go to file

Ctrl + Alt + Shift + Up/Down Arrow: Select and insert the same thing on multiple lines at the same time. Big time saver. Interestingly enough, to use the mouse you have to omit the CTRL key. It's ALT + Shift + Mouse Click.

Ctrl + J: Hides the lower panel.

Ctrl + K M: Change language on current tab. (Ctrl + K, release all keys, then press M)

F8: Run currently selected code (On PowerShell after the extension is installed)

Ctrl + B: Show or hide the sidebar.

Each year I try to wipe out as much of my Azure environment and rebuild it as I possibly can. I like to keep up on product updates that I may have missed, try out new services that I haven't gotten around to just yet, learn new PowerShell commands, and if I'm being honest, organize my environment because I'm a little crazy about things being tidy. Most of the time I do this in July as that's the "slow" time of the year for Microsoft. I'm a little late because there is no real slow time but with Ignite being this week as I get started I thought it would be good timing. Most of the time I remove most of my environment and rebuild. I'll update servers that I want to keep around, I'll clean up my storage account, etc. This year I took a slightly different approach...

I just deleted everything. All of it.

Sometimes you just need to get a fresh start. With this fresh start I thought I would maybe try something new too. I am going to see about rebuilding my entire environment using PowerShell and I'm going to post the journey here. I don't have access to everything that my customers use (I'm not an admin on Microsoft's Azure AD, for instance) but I have enough to be able to post about some of the pitfalls that you may face. This will hopefully also serve as a guide for others to build out a sandbox environment for your own learning purposes.

You don't need to do everything the same way I'm going to do it. I may not use best practices everywhere because I either don't have access or I'm just taking shortcuts, so don't take what I'm doing here and apply it to your production environments at all. I may get part of the way down the road and decide I want to change something too.

Check back with me over the next few weeks as I build things out. Topics will include, but will not be limited to, but also may change, but also might shift around:

1. Networking
2. IaaS Domain
   1. Domain Controller
   2. Workstation
   3. SQL VM(s)
3. SQL
   1. Single Database
   2. Managed Instance
4. Storage Accounts
5. Various Data PaaS services for random demos

I have decided I'm going to work on getting more familiar with Visual Studio Code through this whole process as well. I use it for an editor but I don't use it for PowerShell at all, and I don't use it to connect to or run SQL. I'm very comfortable in the PowerShell ISE and SQL Server Management Studio but I want to use the newer tools more, so I'm forcing myself to learn them now! Also, with the announcement of Visual Studio Online this week, it's a great time to get started in VS Code or just spin up an environment online and get started!

Regardless of which route you decide to go, if you follow me on this Visual Studio adventure as opposed to the PowerShell/PowerShell ISE route, then you will want to install the PowerShell extension. There is also a function that annoys the mess out of me where if you run a line of code the cursor automatically goes down to the console rather than staying up on your script. So I always have to then click back up top. No thank you. So I added a line to my settings.json file to make sure that doesn't happen.

    "powershell.integratedConsole.focusConsoleOnExecute": false

I think that about covers it for today. Check back soon to see what kind of things I'm building and follow along in your environment!