Blogs on KendraLittle.com

Bad News, DBAs, We Are All Developers Now

help@KendraLittle.com (Kendra Little) — Mon, 09 Feb 2026 00:00:00 +0000

I sometimes joke that I’m a Junior Developer and a Principal Database Administrator, which is why I have a Staff level title. I’m not sure it’s a joke anymore, though.

I’m still an OLTP database specialist, but increasingly I’m also a programmer who commits code in multiple languages. I own pull requests that I ship in a release process, following all the same rules and patterns as other developers. My PRs feature unit tests, integration tests, feature flags, the whole shebang.

Tech roles, including DBA roles, are changing rapidly due to AI coding advances. Don’t use AI to write production SQL code without incredibly heavy review and revisions, but do use it to accelerate shipping your improvements to SQL code.

A Pivotal Hackathon

This shift happened pretty suddenly for me. We had a hackathon at work where some members of our platform team helped everyone who didn’t have a working local dev environment get one running – people managers, Product Managers, everyone. I got things working in just over an hour with the help of the team and a couple of AI agents. I was shocked at how much I could suddenly do, see, and learn in my local environment.

This experience made me realize there was nothing standing in the way of me owning all my code changes completely, even the portions that aren’t SQL code. In that hour, I could see the full stack, understand how my SQL changes connected to the application layer, and make changes across both. It was a revelation.

Before the hackathon, I would identify a problem, identify a change in code or an index, then work with a developer on the application code changes, testing, and feature flag work to get the change shipped. This was always a partnership, which was good. This built relationships with developers. But it was often slow: teams have lots of priorities to manage.

In some ways I worried that owning my own changes fully would mean more separation from dev teams. But actually, shifting things to own my own changes and participating more actively in the code review process on both sides has increased the amount of collaborative time I spend with software engineers.

After the hackathon, I realized I can own nearly all of the performance tuning changes I work on end-to-end. I am increasingly proficient at reading and analyzing C# and understanding the common patterns we use in our codebase, and I learn more with every PR both from AI agent feedback as well as from teammates in code review.

Help from Teammates

For the first couple of PRs that I managed on my own, it was helpful to have a partner on the app dev side to help provide feedback early about my code changes. I was overcomplicating some things, and feedback helped me learn to prompt AI agents better to follow our patterns and practices more simply.

Honestly, the hardest part was getting over my own worries that I was going to mess something up. But practices such as feature flagging and testing, along with my teammates’ code reviews and careful testing both in my local dev environment and load testing environments, helped me gain more confidence in my ability to ship more significant changes and refactor more complex code.

In the past I’ve largely focused on making the most impactful changes I can while taking on the smallest amount of risk where possible. This is still a very useful approach, but sometimes you need to take on a larger problem where there is a high amount of risk that you need to minimize as much as possible: I’m now much better equipped to do that on my own.

Benefits of Owning Changes End to End

I really like shipping my own changes. When I own the code changes, I can iterate faster. I can identify the issue, write the fix, test it, and ship it without waiting for someone else’s schedule to align with mine. This speed matters when you’re dealing with performance problems that are impacting customers.

I have increased understanding of the user experience and how slow or failing code impacts customers. I’m learning more about the developer experience of writing features with SQL code, and what’s easy to do versus hard to do. I can see the full picture, from the database query to the application code to how it impacts the user.

The partnership with developers was valuable, and I still have that. But now I can move faster when I need to, and the collaboration happens more through code review on both sides rather than through handoffs.

AI for App Code: Following Patterns Works Extremely Well

This is where AI really shines for database professionals who are expanding their development skills. AI is quite well trained at generating code with languages like C# and Python, which makes owning all my code changes practical.

I’ve found a few things that work well when using AI with app code. Instruct the AI to look at patterns in the repo. Provide example pull requests for reference on patterns to follow. Work with teammates to continuously improve instructions for agents to help developers with common workflows.

AI also helps me considerably in managing and using development environments. Dev environments are increasingly complex with Kubernetes and lots of orchestration. AI helps me understand each environment, troubleshoot failures, and learn how to effectively test with it. It helps me identify how to click-test or script-test code changes, which helps me identify where SQL is called from on the application side and ensure that testing coverage reaches them all.

Where AI is Terrible for SQL Code

There are some problems with having AI generate SQL code and shipping it without heavy review and testing.

Syntax often doesn’t work. AI tooling typically doesn’t have a sandbox environment to validate the specific dialect of SQL code you’re using. It might mix in some PostgreSQL syntax when you are working with SQL Server, or it might use features that don’t exist in your version. It still often hallucinates syntax that doesn’t exist in any database I can find.

You’ll often get incorrect results. AI will generate queries that look reasonable but return wrong data, or that look unreasonable and also return the wrong data. (To be fair, the same thing can be said about my queries.)

Performance will be terrible. Even if AI has the database schema you’re working with, unless you’ve set it up with a test environment with a realistic production dataset, it will not know how a query will be optimized. How could it? I’ve been working with SQL Server for 20 years and I can’t do this either. Query optimizers are complex, and the rules they apply to a query vary based on many factors, including data sizes, data distribution, and the resources assigned to the production instance.

Where AI is Fantastic for SQL Code

Even with its flaws, AI is a huge help at those “heavy review and testing” tasks when it comes to SQL code, whether you generated that code or wrote it yourself. This is simply a matter of having a helper to generate fairly basic procedural scripts following your instructions.

I frequently use AI to:

Generate simple tests that prove the code does what it needs to do
Generate complex tests that compare output of different versions of code to verify the output is correct
Compare the performance of different versions of code to confirm a tuned version is actually faster with realistic data
Ensure parameter sniffing-safe code by generating scripts or engineering a test harness to run code in different orders and cache plans at different points
Research production data distribution by generating queries that help understand why queries perform differently in some environments rather than others

All of these things can be done without AI tooling, of course. AI tooling makes it much more efficient to do this on a regular basis, and to think about ways to write tooling to build this functionality into the developer workflow itself.

Code Review Makes You Better and Builds Collaboration

I should mention that humans are reviewing all my code. While AI can be quite good at application languages, especially when it’s following common patterns, human review is still critical.

Asking others for code reviews also helps remind me that I should be making time on a daily basis for doing code reviews. Code review is a skill that improves with practice. The more PRs I write, the better I get at reviewing other people’s code. I learn what to look for, what questions to ask, and how to provide helpful feedback.

This bidirectional code review process has actually increased the amount of collaborative time I spend with software engineers. Instead of collaborating through handoffs and explanations, we collaborate through code review, which happens more frequently and on a more equal footing.

App Code Projects I’m Thinking About

The hackathon realization that I could own all my code changes opened up possibilities I hadn’t considered before. Now that I’m owning code changes completely, I have several code projects I’m thinking about that would help with database work and observability. These are projects that would have been much harder to pursue when I had to coordinate with developers for every application code change.

Query tagging for observability. I want to tag queries for better observability and improved analysis by code region. I want to automatically inject comments into code, including ORM code, to make it easier to tie queries back to the user experience and build analytics on that. I particularly want to use this to increase the visibility of “clusters” of ORM code that are related, but where queries have unique syntax. Owning both the SQL and application code means I can implement this tagging consistently across the stack.

Flexible load-testing datasets. I’m interested in identifying characteristics of different dataset “flavors” that tend to generate different styles of query plans for customers, and building out flexible load-testing style datasets from them for improved load testing and code validation. One of the historical problems with keeping safe, sanitized or generated performance testing datasets up to date has always been the time and effort it requires. AI offers options for new workflows where it takes on a lot of the toil. Being able to write the scripts and tooling myself means I can iterate on this more quickly.

Multiple query plan paths. I’m thinking about forcing multiple query plan “paths” for very frequent app queries by injecting a static comment value into some of the query strings. SQL Server’s ability to cache plans across sessions conserves memory and CPU, which is great, but for queries that execute at a very high frequency I don’t necessarily want a single query plan in production. If that plan regresses, it impacts everything calling it. I’m thinking about using comments injected into the SQL string at the app layer to help manage this in some scenarios so that I can still get a lot of query plan reuse without having all eggs in one basket. I like the idea of being able to specify having N different query plans per application server, but need to do more research to find the right design to test.

What This Means for Your Career

Database roles are changing. The days of being purely a DBA who only touches SQL Server Management Studio are fading, even for stubborn creatures like me who are most comfortable when curled up snugly inside a complex query execution plan. I don’t have any regrets about the past or not having started out doing more application development – there has been plenty to keep me busy with databases. But, at this point, the barriers are low enough that it doesn’t make sense to NOT become more of a developer.

The hackathon experience showed me that the barrier to owning all your code changes isn’t technical skill. It’s having a working dev environment where it’s safe for you to make mistakes and realizing that nothing is actually stopping you. AI agents make it practical to write application code even if you’re not an expert in C# or Python.

Embracing development skills makes you more valuable. You can ship improvements faster. You can own problems completely. You can understand the full stack, from the database to the application to the user experience.

This doesn’t mean you need to become a full-stack developer. It means you should be comfortable with these skills:

Skill	Why It Matters
Reading and critiquing code in multiple languages	Enables you to own changes across the stack
Using version control	Essential for collaboration and code review
Effective prompting to write tests and the skills to review them for correctness	Ensures your changes work correctly
Shipping code through a release process	Required for production deployments

The bad news is that we’re all developers now. The good news is that it doesn’t suck.

Adaptive Joins and Memory Grants in SQL Server

help@KendraLittle.com (Kendra Little) — Mon, 26 Jan 2026 00:00:00 +0000

Adaptive joins let the optimizer choose between a Hash Join and a Nested Loop join at runtime, which can be fantastic for performance when row count estimates are variable. Recently, when Erik Darling taught two days on TSQL at PASS Community Data Summit, a student asked why a query plan where an adaptive join used a Nested Loop at runtime ended up with a large memory grant anyway.

I didn’t remember the answer to this, but the great thing about co-teaching is that Erik did: adaptive joins always start executing as Hash Joins, which means they have to get memory grants upfront. Even if the query ultimately switches to a Nested Loop at runtime, that memory grant was already allocated. This has real implications for memory usage, especially in high-concurrency environments.

What Are Adaptive Joins?

Adaptive joins were introduced in SQL Server 2017. This join allows the query optimizer to choose between a batch mode Hash Join and a row mode Nested Loops Join at runtime, based on the actual number of rows processed in the Hash Join build phase.

This is a departure from traditional query optimization, where the optimizer makes a one-time decision during plan compilation. With adaptive joins, the optimizer can “change its mind” during execution if the row counts it encounters indicate that a hash join isn’t needed and it can do something more lightweight for that execution of the query.

I’m going to stick to the basics in this post, but there are some excellent deep dives on how adaptive joins work listed at the end of this post.

Understanding Hash Join Build and Probe Phases

Let’s talk about how Hash Joins work. Hash Joins have two phases: build and probe.

Imagine Erik and I are giving a training event. We hire a person to stand at the door and check people in. They know exactly who should be there, and they’re a lot tougher than they look.

Build phase: Before the event starts, we give the door person a list of all registered attendees. The information is in an Excel table (the world’s most popular database). In SQL Server terms, we have built a hash table in memory of all the people who are allowed in.

Probe phase: As people arrive, the door person checks each person’s name against the list– they probe the list to see if each person’s name has a matching value in the hash table. If your name is on the list, they let you through. If not, you’re turned away.

In a hash join, SQL Server does the same thing:

Build phase: Takes one incoming dataset (usually the smaller one) and builds a hash table in memory, organizing rows by their join key
Probe phase: Takes the other dataset it is joining and, for each row, looks up the join key in the hash table to find matching rows

The build phase happens first and requires memory to store the hash table. The probe phase happens second and uses that hash table to quickly find matches / check who is “on the list.”

All Adaptive Joins Are Born as Hash Joins

Key points to remember are:

Hash joins require memory grants upfront - The optimizer needs to allocate memory for the hash table before execution begins
Row mode apply operations don’t need memory grants - They can work with minimal memory, processing rows one at a time
The decision point happens after the build - Once the build side of the hash join completes, SQL Server evaluates whether to continue with the batch mode hash join or switch to a row mode apply

As Paul White emphasizes in his article on the adaptive join threshold:

One thing I want you to bear in mind throughout this piece is that an adaptive join always starts executing as a batch mode hash join. This is true even if the execution plan indicates the adaptive join will transition to apply.

Even if the adaptive join ultimately switches to a row mode apply strategy (which is more efficient for small row counts), it still had to start as a hash join and request the memory grant upfront. The threshold calculation is based on cost estimates and the actual row count from the build phase.

Memory Grant Implications

This is where adaptive joins get interesting from a resource management perspective.

Every execution of a query with an adaptive join must request a memory grant for the hash join, even if the query ultimately switches to a row mode apply at runtime. This follows directly from Paul White’s point that adaptive joins always start executing as batch mode hash joins - the memory grant is requested before SQL Server knows whether it will need to switch strategies.

This means:

Memory grants are requested on every execution - Even if the adaptive join switches to row mode apply, the memory grant was already allocated for the hash join
Overall memory usage can increase - If you have many queries using adaptive joins, your total memory grant usage will be higher than if those queries used row mode apply directly
Memory pressure considerations - In environments with limited memory or high concurrency, adaptive joins can contribute to memory pressure

When to Optimize with Temp Tables and Dynamic SQL

Because adaptive joins always request memory grants, sometimes it’s still worth optimizing queries using temp tables and dynamic SQL to:

Simplify optimization - Break complex queries into simpler parts that the optimizer can handle better
Reduce overall memory grants - Smaller, simpler queries may not need adaptive joins and can use more efficient join strategies that don’t require memory grants
Improve plan stability - Temp tables with statistics can lead to more predictable plans

This doesn’t mean you should avoid adaptive joins entirely - they’re a powerful feature that often improves performance. But understanding the memory grant implications helps you make informed decisions about query optimization strategies.

Example Query with Adaptive Join

Here’s an example query using the StackOverflow2013 database that gets an Adaptive Join on my laptop instance running SQL Server 2025.

USE StackOverflow2013;
GO

ALTER DATABASE CURRENT SET COMPATIBILITY_LEVEL = 150;
GO

EXECUTE dbo.DropIndexes;
GO

CREATE NONCLUSTERED INDEX
 ix_Posts_CreationDate_Score
ON dbo.Posts
(
 CreationDate,
 Score
)
INCLUDE
(
 OwnerUserId,
 PostTypeId,
 CommentCount
)
WHERE PostTypeId IN (1, 2)
WITH
(
 DATA_COMPRESSION = PAGE,
 SORT_IN_TEMPDB = ON
);
GO

CREATE OR ALTER PROCEDURE
 dbo.Adaptive
(
 @Start datetime,
 @End datetime
)
AS
BEGIN
 SELECT
 u.DisplayName,
 u.Reputation,
 p.Score,
 p.CreationDate
 FROM dbo.Posts AS p
 JOIN dbo.Users AS u
 ON u.Id = p.OwnerUserId
 WHERE p.PostTypeId IN (1, 2)
 AND p.CreationDate >= @Start
 AND p.CreationDate < @End
END;
GO

/* This generates a plan with an adaptive join */
EXECUTE dbo.Adaptive '20130101', '20130514';
GO

/* This reuses the plan and chooses to use the nested loop strategy
on the adaptive join */
EXECUTE dbo.Adaptive '20130101', '20130102';
GO

The actual query plan for the second execution looks like this:

Memory grant information is not available in detail per operator, but you can see how much was granted for the query. In this case it is on the SELECT operator.

Detail on how much of the memory grant was used and what was involved in “negotiating” the request is visible in the properties pane for the operator or in the XML of the plan.

Useful Adaptive Join Query Plan Operator Properties

When you look at properties of an adaptive join operator, you’ll see:

Property	Description
Actual Join Type	Whether it picked nested loops or a hash join
Adaptive Threshold Rows	The cardinality at which it switches between nested loops or a hash join
Hash Keys Build	The column(s) from the build input that are hashed into buckets to do the join
Hash Keys Probe	The columns(s) from the probe or nested loop (depending on what is selected) that are used to do the join

Limitations of Adaptive Joins

Adaptive joins have some limitations:

Edition requirements: Batch mode adaptive joins require Enterprise Edition (or Developer/Evaluation editions which include Enterprise features). They’re not available in Standard Edition, Web Edition, or Express Edition.
Compatibility level requirements: Database compatibility level 140 (SQL Server 2017) or higher is required.
Batch mode only: Adaptive joins only work with batch mode execution, which requires either:
- A columnstore index, or…
- Batch Mode on Row Store (SQL Server 2019+, Enterprise Edition only, compatibility level 150 or higher).
Limited join transitions: Adaptive joins only support transitions from batch mode hash joins to row mode apply (correlated nested loops) joins. They don’t support transitions to other join types like merge joins or row mode hash joins.
Threshold calculation sensitivity: The adaptive join threshold depends on the original cardinality estimate of the build input. As Paul White explains, this can lead to parameter sensitivity issues where the same cached plan behaves differently based on the initial cardinality estimate used during optimization.
Query structure requirements: The query must qualify for batch mode execution, and the join must be eligible for the adaptive join optimization. Joins that would require a Key Lookup on the inner side are not eligible. Queries with certain constructs like CROSS APPLY with TOP or OUTER APPLY may not qualify. As Erik Darling notes in his detailed answer, there’s an Extended Event called adaptive_join_skipped that can help you track when and why adaptive joins are skipped, if you like rocket science.
Not supported in natively compiled modules: Natively compiled T-SQL modules (In-Memory OLTP) don’t support adaptive joins.
Memory overhead: As we’ve discussed, adaptive joins always request memory grants for the hash join, even if they switch to row mode apply at runtime.

Enabling and Disabling Adaptive Joins

Adaptive joins are enabled by default when the requirements are met, but you can control them at the database or query level.

Database scoped configuration (affects all queries in the database):

SQL Server 2017:
- To disable: ALTER DATABASE SCOPED CONFIGURATION SET DISABLE_BATCH_MODE_ADAPTIVE_JOINS = ON; (yes, ON means disable - it’s a double negative)
- To re-enable: ALTER DATABASE SCOPED CONFIGURATION SET DISABLE_BATCH_MODE_ADAPTIVE_JOINS = OFF;
SQL Server 2019+:
- To disable: ALTER DATABASE SCOPED CONFIGURATION SET BATCH_MODE_ADAPTIVE_JOINS = OFF;
- To re-enable: ALTER DATABASE SCOPED CONFIGURATION SET BATCH_MODE_ADAPTIVE_JOINS = ON;

Query hint (affects only the specific query):

OPTION (USE HINT('DISABLE_BATCH_MODE_ADAPTIVE_JOINS'));

The query hint takes precedence over the database scoped configuration setting.

Learn More

Adaptive joins are a powerful feature that can improve query performance by choosing the right join strategy at runtime.

However, they ain’t free: they come with memory grant implications that are important to understand, especially in high-concurrency environments or systems with limited memory.

The fact that adaptive joins always start as batch mode hash joins means they always request memory grants, even if they ultimately switch to row mode apply. These joins can be fantastic, but you probably don’t want them running all the time in a highly concurrent system.

Read Paul White’s deep dive on the adaptive join threshold
Watch Erik Darling’s video on adaptive joins in SQL Server
Check out Hugo Kornelis’ page explaining the adaptive join operator

Carrying Baggage Through Query Plans: Why Wide Queries Get Heavy

help@KendraLittle.com (Kendra Little) — Mon, 19 Jan 2026 00:00:00 +0000

I see this pattern repeatedly: a “wide” query that returns many columns and less than 100k rows runs slowly. SQL Server gets slow when it drags large amounts of baggage through the entire query plan, like a solo traveler struggling with massive suitcases in an airport instead of picking them up close to their destination.

SQL Server often minimizes data access by grabbing all the columns it needs early in query execution, then doing joins and filters. This means presentation columns get picked up early.

🔥 Presentation columns: This is a term I picked up from Erik Darling. It refers to descriptive columns that you need in the final result set, but you aren't joining or filtering on them.

This pattern of picking up data baggage early and dragging it through the plan is also one of the reasons that SQL Server loves memory like:

A raccoon loves garbage
A penguin loves a good waddle
A DBA loves to say “No”
A MERGE statement loves chaos
I love a stationary aisle

tldr

The problem: Presentation columns you don’t need for filtering or sorting flow through expensive operations like sorts and hash joins. This frequently causes large memory grants, tempdb spills, and slow queries. Sometimes this is fine. Particularly for wide queries, it often isn’t.

The solution: Separate the core data needed for filtering and sorting from the presentation columns. Two patterns work well when you need to handle a lot of “baggage” in a SQL Server query:

Temp tables: Filter and sort with just the key columns, store your resulting core dataset in a temp table, then join back to get presentation columns. Best for complex logic and when you need statistics on an intermediate object.
Derived tables: Use a nested derived table structure to keep the narrow dataset separate until the final join. If your query uses TOP, push that into the derived table.

🤔 But why? I suspect a major reason SQL Server makes these choices is that it has a horror of random IO, which still haunts it back from the days of rotating hard drives. I wish we had something like the random page cost setting in Postgres to test with, as I'm curious how it would influence this. But we have other ways to tune.

Why this is Often Inefficient

If the query needs to sort data (maybe for a merge join or an ORDER BY clause) or use hash joins for larger datasets, SQL Server must allocate memory to handle the data. And here’s the thing: it allocates memory for ALL the columns it has, not just the keys needed for sorting or joining.

It can’t just leave the presentation columns behind– it has to keep the data together as it flows through the plan.

So the more columns you’re pulling back in your SELECT list, the heavier and slower these operations tend to get. SQL Server estimates memory grants based on the number of rows it expects to process and the width of all columns in those rows. Large memory grants “steal” memory from the buffer pool (it has to come from somewhere). Concurrent queries with large memory grants can lead to RESOURCE SEMAPHORE waits, which can cause other queries to queue before they even get started.

Let’s Work Through an Example

Spoiler, summary of results from below.

Approach	Execution Time	Memory Grant	Max Used Memory	Best For
Original query	13 seconds	15 GB	10.5 GB	Letting the query optimizer make its own choices.
Temp table	3 seconds	6.8 GB (first query), 6.5 MB (second query)	776 MB (first query), 0 (second query)	Complex logic, need statistics, debugging
Derived table	3 seconds	7.8 GB	784 MB	You want to keep everything in one query and aren’t concerned about needing statistics between the steps.

Here is the compat level and indexes I’m using. This was run against SQL Server 2025 on my laptop.

USE StackOverflow2013;
GO

ALTER DATABASE CURRENT SET COMPATIBILITY_LEVEL = 170;
GO

EXECUTE dbo.DropIndexes;
GO

CREATE NONCLUSTERED INDEX
 ix_Posts_OwnerUserId_Score_PostTypeId
ON dbo.Posts
(
 OwnerUserId,
 PostTypeId,
 Score
)
WITH
(
 DATA_COMPRESSION = PAGE,
 SORT_IN_TEMPDB = ON
);
GO

CREATE NONCLUSTERED INDEX
 ix_Comments_UserId_Score
ON dbo.Comments
(
 UserId,
 Score
)
WITH
(
 DATA_COMPRESSION = PAGE,
 SORT_IN_TEMPDB = ON
);
GO

CREATE NONCLUSTERED INDEX
 ix_Comments_PostId
ON dbo.Comments
(
 PostId
)
WITH
(
 DATA_COMPRESSION = PAGE,
 SORT_IN_TEMPDB = ON
);
GO

A Wide Query Enters the Chat

I’ve titled this query “StackOverflow Sick Burns and Top Pithyness.”

The query finds high-scoring comments, where:

Comment has Score > 200 (required)
Comment is NOT on a post the user created themselves
If user has questions, the comment score must be higher than their highest scoring question
Returns top 100 comments (by score) with user details, comment details, post details, and user’s highest scoring question (if any)
Ensures one question per user (highest score, highest Id as tie-breaker) to prevent duplicate rows

The query takes 13 seconds on my laptop.

SELECT TOP 100
 /* Core columns used for filtering/sorting */
 userId = u.Id,
 commentScore = c.Score,
 questionScore = userHighestQuestion.questionScore,
 scoreDifference = c.Score - ISNULL(userHighestQuestion.questionScore, 0),

 /* Presentation-only: Comment details */
 commentId = c.Id,
 commentCreationDate = c.CreationDate,
 commentText = c.Text, /* large varchar - presentation only */

 /* Presentation-only: User details */
 userDisplayName = u.DisplayName,
 userReputation = u.Reputation,
 userLocation = u.Location,
 userCreationDate = u.CreationDate,
 userUpvotes = u.UpVotes,
 userDownvotes = u.DownVotes,
 userViews = u.Views,
 userAboutMe = u.AboutMe, /* LOB column - presentation only */

 /* Presentation-only: Post that comment is on */
 commentPostTitle = p.Title,
 commentPostBody = p.Body, /* LOB column - presentation only */
 commentPostViewCount = p.ViewCount,
 commentPostScore = p.Score,
 commentPostTags = p.Tags,
 commentPostOwnerDisplayName = up.DisplayName,
 commentPostOwnerReputation = up.Reputation,

 /* Presentation-only: User's highest question (if any) */
 questionId = q.Id,
 questionTitle = q.Title,
 questionCreationDate = q.CreationDate,
 questionViewCount = q.ViewCount,
 questionTags = q.Tags,
 questionBody = q.Body, /* LOB column - presentation only */

 /* Presentation-only: Calculated columns */
 daysSinceComment =
 DATEDIFF
 (
 DAY,
 c.CreationDate,
 GETDATE()
 ),
 userReputationCategory =
 CASE
 WHEN u.Reputation < 100
 THEN N'New User'
 WHEN u.Reputation < 1000
 THEN N'Regular User'
 WHEN u.Reputation < 10000
 THEN N'Established User'
 ELSE N'Expert User'
 END
FROM dbo.Comments AS c
JOIN dbo.Users AS u
 ON u.Id = c.UserId /* excludes comments with NULL UserId (deleted users) */
JOIN dbo.Posts AS p
 ON p.Id = c.PostId
JOIN dbo.Users AS up
 ON up.Id = p.OwnerUserId /* excludes comments on community wiki posts (NULL OwnerUserId) */
LEFT JOIN
(
 /* Get one question per user: highest score, highest Id as tie-breaker */
 SELECT
 userId = q.OwnerUserId,
 questionId = q.Id,
 questionScore = q.Score
 FROM dbo.Posts AS q
 WHERE q.PostTypeId = 1
 /* Find questions with the maximum score for this user, highest Id as tie-breaker */
 AND q.Id =
 (
 SELECT
 MAX(qMaxScore.Id)
 FROM dbo.Posts AS qMaxScore
 WHERE qMaxScore.OwnerUserId = q.OwnerUserId
 AND qMaxScore.PostTypeId = 1
 AND qMaxScore.Score =
 (
 SELECT
 MAX(qUser.Score)
 FROM dbo.Posts AS qUser
 WHERE qUser.OwnerUserId = q.OwnerUserId
 AND qUser.PostTypeId = 1
 )
 )
) AS userHighestQuestion
 ON userHighestQuestion.userId = c.UserId
LEFT JOIN dbo.Posts AS q
 ON q.Id = userHighestQuestion.questionId
WHERE
 CASE
 WHEN c.Score > 200
 AND c.UserId <> p.OwnerUserId
 AND (userHighestQuestion.questionScore IS NULL OR c.Score > userHighestQuestion.questionScore)
 THEN 1
 ELSE 0
 END = 1
ORDER BY
 c.Score DESC;
GO

Here is the overall shape of the actual query plan:

The plan does some interesting things, but one choice that goes badly is that it picks up presentation columns Body, Tags, Title, and more from Posts for the table aliased q. It then feeds all that data into a Filter operator.

This screenshot is from a different execution than the one in the plan shape above. In this execution the filter operator took even longer, but in both cases they output the same columns.

The Body column is the whole text of questions – that’s a lot of data. The query has a 15 GB memory grant, and it uses 10.5 GB of that.

Tuning Option: Temp Table

Let’s break the query into two steps.

First, get the core data for filtering and joining into a temp table with just the key columns and reduce down to our 100 rows.
Then join back to get the presentation columns.

I am assuming I don’t need to order results in the second query and that ordering can be handled in the presentation layer, but ordering is required where we process TOP.

This returns in 3 seconds on my laptop. The memory grant for the query that populates the temp table is 6.8 GB (it used 776 MB) and 6.5 MB for the query that reads from the temp table and joins to get detail data (it used 0).

The main concept of this tuning approach is that we are specifying that we want to establish our “core” dataset and pop it into a temporary table before we fetch columns for presentation. SQL Server loves a temporary table because it can automatically create column statistics on it, which also can help it optimize the second query well.

DROP TABLE IF EXISTS #core;

SELECT TOP 100 p.Id as PostId, u.Id as UserId, c.Id as CommentId, q.Id as questionId, userHighestQuestion.questionScore, up.DisplayName, up.Reputation
INTO #core
FROM dbo.Comments AS c
JOIN dbo.Users AS u
 ON u.Id = c.UserId /* excludes comments with NULL UserId (deleted users) */
JOIN dbo.Posts AS p
 ON p.Id = c.PostId
JOIN dbo.Users AS up
ON up.Id = p.OwnerUserId /* excludes comments on community wiki posts (NULL OwnerUserId) */
LEFT JOIN
(
 /* Get one question per user: highest score, highest Id as tie-breaker */
 SELECT
 userId = q.OwnerUserId,
 questionId = q.Id,
 questionScore = q.Score
 FROM dbo.Posts AS q
 WHERE q.PostTypeId = 1
 /* Find questions with the maximum score for this user, highest Id as tie-breaker */
 AND q.Id =
 (
 SELECT
 MAX(qMaxScore.Id)
 FROM dbo.Posts AS qMaxScore
 WHERE qMaxScore.OwnerUserId = q.OwnerUserId
 AND qMaxScore.PostTypeId = 1
 AND qMaxScore.Score =
 (
 SELECT
 MAX(qUser.Score)
 FROM dbo.Posts AS qUser
 WHERE qUser.OwnerUserId = q.OwnerUserId
 AND qUser.PostTypeId = 1
 )
 )
) AS userHighestQuestion
 ON userHighestQuestion.userId = c.UserId
LEFT JOIN dbo.Posts AS q
 ON q.Id = userHighestQuestion.questionId
WHERE
 CASE
 WHEN c.Score > 200
 AND c.UserId <> p.OwnerUserId
 AND (userHighestQuestion.questionScore IS NULL OR c.Score > userHighestQuestion.questionScore)
 THEN 1
 ELSE 0
 END = 1
ORDER BY
 c.Score DESC;

SELECT
 /* Core columns used for filtering/sorting */
 userId = u.Id,
 commentScore = c.Score,
 questionScore = core.questionScore,
 scoreDifference = c.Score - ISNULL(core.questionScore, 0),

 /* Presentation-only: Comment details */
 commentId = c.Id,
 commentCreationDate = c.CreationDate,
 commentText = c.Text, /* large varchar - presentation only */

 /* Presentation-only: User details */
 userDisplayName = u.DisplayName,
 userReputation = u.Reputation,
 userLocation = u.Location,
 userCreationDate = u.CreationDate,
 userUpvotes = u.UpVotes,
 userDownvotes = u.DownVotes,
 userViews = u.Views,
 userAboutMe = u.AboutMe, /* LOB column - presentation only */

 /* Presentation-only: Post that comment is on */
 commentPostTitle = p.Title,
 commentPostBody = p.Body, /* LOB column - presentation only */
 commentPostViewCount = p.ViewCount,
 commentPostScore = p.Score,
 commentPostTags = p.Tags,
 commentPostOwnerDisplayName = core.DisplayName,
 commentPostOwnerReputation = core.Reputation,

 /* Presentation-only: User's highest question (if any) */
 questionId = q.Id,
 questionTitle = q.Title,
 questionCreationDate = q.CreationDate,
 questionViewCount = q.ViewCount,
 questionTags = q.Tags,
 questionBody = q.Body, /* LOB column - presentation only */

 /* Presentation-only: Calculated columns */
 daysSinceComment =
 DATEDIFF
 (
 DAY,
 c.CreationDate,
 GETDATE()
 ),
 userReputationCategory =
 CASE
 WHEN u.Reputation < 100
 THEN N'New User'
 WHEN u.Reputation < 1000
 THEN N'Regular User'
 WHEN u.Reputation < 10000
 THEN N'Established User'
 ELSE N'Expert User'
 END
FROM #core as core
JOIN dbo.Posts as p on core.PostId=p.Id
JOIN dbo.Comments as c on core.CommentId=c.Id
JOIN dbo.Users as u on core.UserId=u.Id
JOIN dbo.Posts as q on q.Id = core.questionId;
GO

When to use temp tables: Use this approach when you need column statistics for optimization, have complex multi-step logic that benefits from breaking into stages, or want to debug intermediate results. Temp tables also work well when you need to reuse the core dataset multiple times.

Tuning Option: Derived Table

Do we really need the temp table though?

Let’s see how far we can get while staying in a single query. This tuning option focuses on defining the core dataset in derived tables or CROSS APPLYs, then joining back to get the wider presentation columns. In this example we’re using TOP, so we push that into a derived table and join to it. The same pattern can work for queries without TOP: use the derived table to separate the narrow filtering/sorting dataset from the presentation columns.

This query finishes on my laptop in 3 seconds. The memory grant is 7.8 GB, and it used 784 MB of that. If this was an actual production query I’d figure out how to tone that grant down a bit and tune this more. But for the purpose of a relatively quick post, this makes the general point.

When to use derived tables: Use this approach when you want a single query but can accept higher memory grants (or take the time to tune them away), or when the query runs infrequently enough that the memory grant isn’t a concern.

SELECT
 /* Core columns used for filtering/sorting */
 userId = u.Id,
 commentScore = c.Score,
 questionScore = core.questionScore,
 scoreDifference = c.Score - ISNULL(core.questionScore, 0),

 /* Presentation-only: Comment details */
 commentId = c.Id,
 commentCreationDate = c.CreationDate,
 commentText = c.Text, /* large varchar - presentation only */

 /* Presentation-only: User details */
 userDisplayName = u.DisplayName,
 userReputation = u.Reputation,
 userLocation = u.Location,
 userCreationDate = u.CreationDate,
 userUpvotes = u.UpVotes,
 userDownvotes = u.DownVotes,
 userViews = u.Views,
 userAboutMe = u.AboutMe, /* LOB column - presentation only */

 /* Presentation-only: Post that comment is on */
 commentPostTitle = p.Title,
 commentPostBody = p.Body, /* LOB column - presentation only */
 commentPostViewCount = p.ViewCount,
 commentPostScore = p.Score,
 commentPostTags = p.Tags,
 commentPostOwnerDisplayName = core.DisplayName,
 commentPostOwnerReputation = core.Reputation,

 /* Presentation-only: User's highest question (if any) */
 questionId = q.Id,
 questionTitle = q.Title,
 questionCreationDate = q.CreationDate,
 questionViewCount = q.ViewCount,
 questionTags = q.Tags,
 questionBody = q.Body, /* LOB column - presentation only */

 /* Presentation-only: Calculated columns */
 daysSinceComment =
 DATEDIFF
 (
 DAY,
 c.CreationDate,
 GETDATE()
 ),
 userReputationCategory =
 CASE
 WHEN u.Reputation < 100
 THEN N'New User'
 WHEN u.Reputation < 1000
 THEN N'Regular User'
 WHEN u.Reputation < 10000
 THEN N'Established User'
 ELSE N'Expert User'
 END
FROM
(
 SELECT TOP 100 p.Id as PostId, u.Id as UserId, c.Id as CommentId, q.Id as questionId, userHighestQuestion.questionScore, up.DisplayName, up.Reputation
 FROM dbo.Comments AS c
 JOIN dbo.Users AS u
 ON u.Id = c.UserId /* excludes comments with NULL UserId (deleted users) */
 JOIN dbo.Posts AS p
 ON p.Id = c.PostId
 JOIN dbo.Users AS up
 ON up.Id = p.OwnerUserId /* excludes comments on community wiki posts (NULL OwnerUserId) */
 LEFT JOIN
 (
 /* Get one question per user: highest score, highest Id as tie-breaker */
 SELECT
 userId = q.OwnerUserId,
 questionId = q.Id,
 questionScore = q.Score
 FROM dbo.Posts AS q
 WHERE q.PostTypeId = 1
 /* Find questions with the maximum score for this user, highest Id as tie-breaker */
 AND q.Id =
 (
 SELECT
 MAX(qMaxScore.Id)
 FROM dbo.Posts AS qMaxScore
 WHERE qMaxScore.OwnerUserId = q.OwnerUserId
 AND qMaxScore.PostTypeId = 1
 AND qMaxScore.Score =
 (
 SELECT
 MAX(qUser.Score)
 FROM dbo.Posts AS qUser
 WHERE qUser.OwnerUserId = q.OwnerUserId
 AND qUser.PostTypeId = 1
 )
 )
 ) AS userHighestQuestion
 ON userHighestQuestion.userId = c.UserId
 LEFT JOIN dbo.Posts AS q
 ON q.Id = userHighestQuestion.questionId
 WHERE
 CASE
 WHEN c.Score > 200
 AND c.UserId <> p.OwnerUserId
 AND (userHighestQuestion.questionScore IS NULL OR c.Score > userHighestQuestion.questionScore)
 THEN 1
 ELSE 0
 END = 1
 ORDER BY
 c.Score DESC
 ) as core

JOIN dbo.Posts as p on core.PostId=p.Id
JOIN dbo.Comments as c on core.CommentId=c.Id
JOIN dbo.Users as u on core.UserId=u.Id
JOIN dbo.Posts as q on q.Id = core.questionId;

Here is the overall shape of the query plan:

The use of the derived table forces SQL Server to push that portion of the query into the block to the right, then joins to the other tables are processed.

Do note that I wrote forces: similar to the temp table rewrite, we are making a choice that dictates how this will be processed. We are removing choices from the query optimizer.

Common Pitfalls to Watch For

When splitting queries to reduce baggage, watch out for these issues:

Forgetting join keys: Make sure your narrow dataset includes all the keys you need to join back to get the presentation columns.
NULL handling: When splitting queries, consider how NULLs are handled. The original query might have handled NULLs differently than your split version.
Over-optimizing: If a query runs infrequently, the optimization effort might not be worth it. Focus on queries that run often or are causing performance problems.

Lightening the Load

When SQL Server carries wide columns through expensive operations like sorts and hash joins, it requires significant memory and slows down your queries. By separating the core data you need for filtering and sorting from the presentation columns, you can significantly reduce the memory footprint and improve query performance.

The next time you see a slow query with a wide result set, check the execution plan. Look at operators like Sort or Hash Match and examine their “Output List” or “Defined Values” properties. If you see many more columns than are needed for the operation itself, you’ve found the baggage. Try one of these two approaches to lighten the load.

If the query runs very frequently, it’s worth considering your options for whether application caching might help out, too. As they say, the fastest query is the one you never run.

Testing SQL Server 2025 Resource Governor tempdb Limits with a Query that Spills a Terabyte

help@KendraLittle.com (Kendra Little) — Mon, 12 Jan 2026 00:00:00 +0000

SQL Server 2025 introduces a new Resource Governor capability to manage tempdb usage, along with making Resource Governor available in Standard Edition.

I wondered: can Resource Governor’s new tempdb feature help contain queries that don’t use temporary tables, but which spill massive amounts of data to tempdb? The docs say yes, but I always like to get hands-on experience when I can.

I’ve got an awful query that spills like a soft-serve machine choosing violence. Let’s test drive the new tempdb governance features in SQL Server 2025.

Meet our Test Query Whose Sort Spills a Terabyte of Data

I’m using the StackOverflow2013 database with a query built to spill. The query gets a 15 GB memory grant due to the combination of window functions calculating aggregates across large partitions and multiple LEFT JOINs producing duplicate rows for windowing calculations.

That 15 GB memory grant is far too little for the amount of data going into a sort operator in the query plan, and the memory spill to disk easily fills up my 1 TB of local free space on my laptop if it’s not contained.

First, this script configures database settings and indexes used by the query:

USE StackOverflow2013;
GO

ALTER DATABASE CURRENT SET COMPATIBILITY_LEVEL = 170;
GO

EXECUTE dbo.DropIndexes;
GO

CREATE NONCLUSTERED INDEX
 ix_Posts_PostTypeId_ETC
ON dbo.Posts
([PostTypeId])
INCLUDE ([AcceptedAnswerId],[AnswerCount],[CommentCount],[CreationDate],[LastActivityDate],[OwnerUserId],[Score],[Tags],[Title],[ViewCount])
WITH
(
 DATA_COMPRESSION = PAGE
);
GO

CREATE NONCLUSTERED INDEX
 ix_Comments_PostId
ON dbo.Comments
 (PostId)
INCLUDE
(
 CreationDate,
 Score
)
WITH
(
 DATA_COMPRESSION = PAGE
);
GO

And this script is the spilling query itself:

/* This query is not good for much except tempdb spills. */
USE StackOverflow2013;
GO

SELECT TOP (100)
 QuestionId = q.Id,
 QuestionTitle = q.Title,
 QuestionCreationDate = q.CreationDate,
 QuestionScore = q.Score,
 QuestionViews = q.ViewCount,
 QuestionAnswerCount = q.AnswerCount,
 QuestionCommentCount = q.CommentCount,
 QuestionTags = q.Tags,
 QuestionLastActivityDate = q.LastActivityDate,
 QuestionOwnerId = uq.Id,
 QuestionOwnerName = uq.DisplayName,
 QuestionOwnerReputation = uq.Reputation,
 QuestionOwnerLocation = uq.Location,
 QuestionOwnerCreationDate = uq.CreationDate,
 AcceptedAnswerId = a.Id,
 AcceptedAnswerScore = a.Score,
 AcceptedAnswerCreationDate = a.CreationDate,
 AcceptedAnswerOwnerId = ua.Id,
 AcceptedAnswerOwnerName = ua.DisplayName,
 AcceptedAnswerOwnerReputation = ua.Reputation,
 TotalUpVotes =
 SUM
 (
 CASE
 WHEN v.VoteTypeId = 2
 THEN 1
 ELSE 0
 END
 ) OVER
 (
 PARTITION BY
 q.Id
 ),
 TotalFavorites =
 SUM
 (
 CASE
 WHEN v.VoteTypeId = 5
 THEN 1
 ELSE 0
 END
 ) OVER
 (
 PARTITION BY
 q.Id
 ),
 RecentCommentCount =
 COUNT_BIG
 (
 CASE
 WHEN c.CreationDate >= '2012-01-01'
 THEN 1
 END
 ) OVER
 (
 PARTITION BY
 q.Id
 ),
 QuestionOwnerBadgeCount =
 COUNT_BIG(b.Id) OVER
 (
 PARTITION BY
 uq.Id
 ),
 QuestionOwnerGoldBadges =
 SUM
 (
 CASE
 WHEN b.Name IN ('Legendary', 'Enlightened', 'Great Answer')
 THEN 1
 ELSE 0
 END
 ) OVER
 (
 PARTITION BY
 uq.Id
 )
FROM dbo.Posts AS q
JOIN dbo.Users AS uq
 ON uq.Id = q.OwnerUserId
LEFT JOIN dbo.Posts AS a
 ON a.Id = q.AcceptedAnswerId
LEFT JOIN dbo.Users AS ua
 ON ua.Id = a.OwnerUserId
LEFT JOIN dbo.Votes AS v
 ON v.PostId = q.Id
LEFT JOIN dbo.Comments AS c
 ON c.PostId = q.Id
LEFT JOIN dbo.Badges AS b
 ON b.UserId = uq.Id
WHERE q.PostTypeId = 1
AND CASE
 WHEN uq.Reputation >= 50000
 AND q.Score >= 10
 THEN 1
 WHEN uq.Reputation BETWEEN 5000 AND 49999
 AND q.Score >= 20
 AND q.ViewCount >= 5000
 THEN 1
 WHEN uq.Reputation < 5000
 AND (
 q.Score >= 40
 OR (a.Score IS NOT NULL AND a.Score >= 15)
 )
 THEN 1
 WHEN ua.Reputation >= 20000
 AND v.VoteTypeId = 2
 AND c.CreationDate >= '2012-01-01'
 THEN 1
 ELSE 0
 END = 1
ORDER BY
 q.Score DESC,
 q.ViewCount DESC,
 q.CreationDate DESC;

SQL Server 2025 Resource Governor tempdb Space Governance

SQL Server 2025 introduces Resource Governor tempdb Space Governance, which allows you to set limits in one of two ways.

To constrain tempdb usage there are two “group” level options as described in Create Workload Group:

GROUP_MAX_TEMPDB_DATA_MB: “the maximum amount of space that a workload group can consume in the tempdb data files, in megabytes… The limit is for the total space consumed in tempdb by all sessions in a workload group.”
GROUP_MAX_TEMPDB_DATA_PERCENT: “Specifies the maximum amount of space that a workload group can consume in the tempdb data files, in percent of the maximum tempdb size.” There’s a table in the docs that gets into the configuration requirements for this setting to be in effect. One requirement is that this doesn’t work if you also have GROUP_MAX_TEMPDB_DATA_MB configured for the workload group.

When a query exceeds the configured limit, SQL Server aborts it with error 1138, severity 17, Could not allocate a new page for database 'tempdb' because that would exceed the limit set for workload group 'workload-group-name'.

Configure Resource Governor for our Test

To test tempdb governance with our spilling query, we will:

Create a resource pool
Create a workload group with tempdb limits that uses the resource pool
Create a classifier function to route our test queries to that workload group
Enable Resource Governor
Verify the configuration
Run the test query from a classified session
Observe the tempdb usage of the pool by querying sys.dm_resource_governor_workload_groups

Step 1: Create a Resource Pool

USE master;
GO

CREATE RESOURCE POOL tempdb_spill_test_pool;
GO

A resource pool defines the physical resources available to the group of queries that you classify to use the pool. The resource pool lets you set things like CPU percentage limits with MAX_CPU_PERCENT and memory limits with MAX_MEMORY_PERCENT.

The tempdb governance feature does not have any options on the resource pool itself, the options are all on the workload group. You can use the default pool if you want, but I think it’s useful to create separate pools for different workloads for tracking, and you may wish to apply pool limits at some time in the future.

I’m creating a basic pool without specifying resource limits.

Step 2: Create a Workload Group with tempdb Limits

CREATE WORKLOAD GROUP tempdb_spill_test_group
WITH
(
 GROUP_MAX_TEMPDB_DATA_MB = 10240
)
USING tempdb_spill_test_pool;
GO

The USING clause assigns this workload group to our resource pool. Sessions classified into this workload group get the tempdb GROUP_MAX_TEMPDB_DATA_MB limit (10 GB) and any other resource limits we put on the pool.

Some workload group options allow you to define policies for the whole group, such as GROUP_MAX_REQUESTS. Some options define policies for individual requests within the group. For example, REQUEST_MAX_MEMORY_GRANT_PERCENT and REQUEST_MAX_CPU_TIME_SEC each allow you to limit the resources for individual queries that run in the pool. You can also set the MAX_DOP (maximum degree of parallelism) for all requests in the pool.

🔥 Don't Read Too Much Into This: I'm not saying 10 GB is a "best practice" value for how much space to allow a workload group to use in tempdb. Actual tempdb requirements vary dramatically across production environments. Some workloads need very little, some need a lot. Baseline a specific system to know what's appropriate.

Step 3: Create a Classifier Function

USE master;
GO

CREATE OR ALTER FUNCTION
 dbo.tempdb_test_classifier()
RETURNS sysname
WITH SCHEMABINDING
AS
BEGIN
 DECLARE
 @workload_group_name sysname = N'default';

 IF APP_NAME() = N'tempdb_spill_test'
 BEGIN
 SELECT
 @workload_group_name = N'tempdb_spill_test_group';
 END;

 RETURN @workload_group_name;
END;
GO

A classifier function is a scalar user-defined function that runs for every new connection. It returns the name of the workload group to assign the session to. The function must be created in the master database with SCHEMABINDING.

Only one classifier function can be active on a given SQL Server Instance at a time.

Classifier functions commonly use session properties like APP_NAME(), SUSER_SNAME() (login name), HOST_NAME() (client computer name), or custom logic to route sessions. The function executes during login, so keep it lightweight and fast to avoid impacting performance at login time and to avoid connection timeouts.

This also means that when you enable Resource Governor, sessions that are already connected will not use the pools or workload groups you’ve newly configured. Sessions are only classified the next time they log in.

The classifier function I’m using here checks the application name, because that’s very simple to use for testing.

When a session connects with the application name ’tempdb_spill_test’, it assigns that session to tempdb_spill_test_group.
All other sessions go to the ‘default’ workload group.

Step 4: Configure and Enable Resource Governor

ALTER RESOURCE GOVERNOR
WITH
(
 CLASSIFIER_FUNCTION = dbo.tempdb_test_classifier
);
GO

ALTER RESOURCE GOVERNOR RECONFIGURE;
GO

Step 5: Verify the Configuration

SELECT
 group_name = wg.name,
 wg.group_max_tempdb_data_mb,
 wg.group_max_tempdb_data_percent,
 pool_name = rp.name
FROM sys.resource_governor_workload_groups AS wg
JOIN sys.dm_resource_governor_resource_pools AS rp
 ON rp.pool_id = wg.pool_id
WHERE wg.name = N'tempdb_spill_test_group';
GO

The query results show that our workload group is configured with the 10 GB limit for tempdb data, and that the group is assigned to the resource pool we created as well.

Step 6: Run the Test Query in a Classified Session

To run the test query under resource governance, open a new query window in SSMS and set the application name in the connection properties. If you’re using SQL Server Management Studio 22, here are the steps in the new database connection window:

Select File, New, Database Engine Query
In the connection dialog, select Advanced
Under Context set the application name to: tempdb_spill_test
Select OK
Make sure ServerName and other connection info is set correctly and select Connect

Run this query in your new session to verify that it has been classified correctly:

SELECT
 workload_group_name = wg.name
FROM sys.dm_exec_sessions AS s
JOIN sys.dm_resource_governor_workload_groups AS wg
 ON wg.group_id = s.group_id
WHERE s.session_id = @@SPID;
GO

This should return tempdb_spill_test_group as the workload group name.

Now run the test query above in this session.

The spilling query hits the 10 GB limit on my laptop in around 17 seconds.

When it hits the limit, SQL Server aborts the query and I see the foretold error:

Msg 1138, Level 17, State 1, Line 3
Could not allocate a new page for database 'tempdb' because that would exceed
the limit set for workload group 'tempdb_spill_test_group', group_id 256.

Step 7. Query tempdb Usage and Violations from `sys.dm_resource_governor_workload_groups`

Since the query fails quickly, it’s easy to run it multiple times and watch the workload group’s tempdb usage while it is running:

SELECT
 wg.group_id,
 wg.name,
 wg.tempdb_data_space_kb,
 wg.peak_tempdb_data_space_kb,
 wg.total_tempdb_data_limit_violation_count
FROM sys.dm_resource_governor_workload_groups AS wg
WHERE wg.name = N'tempdb_spill_test_group';
GO

Here’s an example of what the output looks like on my laptop:

The total_tempdb_data_limit_violation_count column does not count the number of queries that failed

This column in sys.dm_resource_governor_workload_groups is defined as:

The number of times a request was aborted with error 1138 because it would exceed the limit on tempdb data space consumption for the workload group.

In some brief testing, I usually see this number increases by 8 on my laptop every time my query hits the limit and is aborted. The query is running at max degree of parallelism 8 on my laptop. Sometimes I see it increment by a different amount, though, such as 5.

Was the effective maxdop lowered for tha run for some reason? I’m not sure, I haven’t looked into it that much, but the increment is definitely not the count of individual query runs that failed.

Cleanup: Remove Resource Governor Configuration

This script undoes what the demo above sets up. You need to close all active sessions that are using the workload group/ resource pool before this will complete successfully.

ALTER RESOURCE GOVERNOR DISABLE;
GO

ALTER RESOURCE GOVERNOR
WITH
(
 CLASSIFIER_FUNCTION = NULL
);
GO

ALTER RESOURCE GOVERNOR RECONFIGURE;
GO

DROP FUNCTION IF EXISTS dbo.tempdb_test_classifier;
GO

DROP WORKLOAD GROUP tempdb_spill_test_group;
GO

DROP RESOURCE POOL tempdb_spill_test_pool;
GO

Did Resource Governance Help with tempdb Spills?

✅ Yes, this Resource Governor feature protected my hard drive from the mad space consumption of this truly terrible query. This confirms what the documentation says: the tempdb space governance feature does not just apply to temp table usage, it also helps contain memory spills in query plan operators.

My takeaways:

This is a great feature to have in SQL Server, particularly if you have large amounts of data and queries can’t always be tested with production dataset sizes before getting released into your SQL Server. If you have auto-generated code or folks who need to rapidly develop queries for analytics, you’re probably pretty excited to get your hands on this feature.
This tempdb governance feature doesn’t provide warnings or controlled degradation. It’s a hard limit: stay under it or queries will fail. This makes it useful for protecting the server, but it means you need to tune your queries or increase memory grants to avoid hitting the limit.
The total_tempdb_data_limit_violation_count column in sys.dm_resource_governor_workload_groups is confusing. The docs read like it counts individual queries that fail, when in fact it seems to count something more like threads that fail when they hit the limit. This doesn’t seem like a huge problem as long as you don’t misinterpret it.

Overall, I think this feature is a welcome addition to Resource Governor, and it’s terrific that Resource Governor is available for Standard Edition in SQL Server 2025.

How to Trace Trigger Executions with Query Store and Extended Events

help@KendraLittle.com (Kendra Little) — Mon, 05 Jan 2026 00:00:00 +0000

Triggers can be tricky to observe in SQL Server. When you need to understand exactly what a trigger is doing and what it impacts, Query Store and Extended Events both have something to offer, but using them effectively requires navigating some confusing nuances. Query Store tracks query-level executions aggregated by query_id, but has a habit of generating a lot of query_ids with different context settings for triggers. Extended Events can capture trigger module starts and individual statement completions within triggers.

This post walks through setting up both observation methods on a test trigger, then compares what each one shows. You’ll see how Query Store data gets spread across multiple query_ids and context_settings_id values, a variety of ways to trace triggers in XEvents, and why the numbers don’t always match up exactly between traces and Query Store.

I’m using the StackOverflow2013 sample database on SQL Server 2025 Developer Edition to create a trigger, test it with various DML operations (INSERT, UPDATE, DELETE, MERGE, and dynamic SQL), then analyze the results from both observation methods.

Setup: Enable and Clear Query Store

First, let’s make sure Query Store is enabled. This clears all data from Query Store so be careful where you run this.

USE StackOverflow2013;
GO

ALTER DATABASE CURRENT
SET QUERY_STORE = ON;
GO

ALTER DATABASE CURRENT
SET QUERY_STORE
(
 OPERATION_MODE = READ_WRITE,
 QUERY_CAPTURE_MODE = AUTO,
 MAX_STORAGE_SIZE_MB = 10240,
 CLEANUP_POLICY = (STALE_QUERY_THRESHOLD_DAYS = 14)
);
GO

/* WARNING: Clear Query Store data for clean testing */
/* This removes all Query Store data - use with caution in production */
ALTER DATABASE CURRENT
SET QUERY_STORE CLEAR;
GO

Setup: Create the Audit Table

Next, create a simple audit table to track changes to the Posts table:

IF OBJECT_ID(N'dbo.AuditPosts', N'U') IS NOT NULL
BEGIN
 DROP TABLE dbo.AuditPosts;
END;
GO

CREATE TABLE
 dbo.AuditPosts
(
 AuditPostsId bigint NOT NULL IDENTITY(1,1),
 PostsId integer NOT NULL,
 Action char(1) NOT NULL,
 ActivityDate datetime NOT NULL,
 ColumnName sysname NOT NULL,
 ColumnValue nvarchar(250) NULL,
 CONSTRAINT PK_AuditPosts_AuditPostsId
 PRIMARY KEY CLUSTERED (AuditPostsId)
);
GO

Setup: Create the Trigger on Posts

Now I’m creating a wild and safety-free trigger on the Posts table. This trigger has no safety checks (no IF EXISTS blocks) and uses UNION ALL with separate queries for each column to handle NULLs.

When we modify the Posts table (INSERT, UPDATE, or DELETE), the trigger automatically fires and performs these actions:

When we modify Posts	Trigger behavior
INSERT or UPDATE	Trigger updates `LastActivityDate` on the Posts table
INSERT	Trigger adds 4 rows to AuditPosts (one for each of Title, PostTypeId, OwnerUserId, Score) with Action = ‘I’, using separate UNION ALL queries that handle NULL values for Title and OwnerUserId
UPDATE	Trigger adds 4 rows to AuditPosts with Action = ‘U’ if any of Title, PostTypeId, OwnerUserId, or Score changed (inserts all 4 rows with “before” values from the deleted table when any column changes)
DELETE	Trigger adds 4 rows to AuditPosts with Action = ‘D’, using separate UNION ALL queries that handle NULL Title for answers

Important: All modifications to AuditPosts come from the trigger. We never directly INSERT, UPDATE, or DELETE from AuditPosts. We only modify Posts, and the trigger handles all AuditPosts changes automatically.

By “safety-free”, I mean this trigger lacks the Good Things in Trigger Life:

It’s not selectively firing.
It does not check if ROWCOUNT_BIG() = 0 and return if nothing happens.
It doesn’t do any of the following:
- Existence checks (no EXISTS or NOT EXISTS clauses) to see if the inserted and deleted tables have rows in them
- Use the UPDATE() trigger function
- Use the COLUMNS_UPDATED() trigger function to test if the trigger should execute certain actions

Instead, it uses only NULL comparisons to distinguish between INSERT, UPDATE, and DELETE operations. For INSERT operations, it joins inserted to deleted using LEFT JOIN and check if deleted.Id IS NULL. For DELETE operations, we check if inserted.Id IS NULL when joining deleted to inserted.

IF OBJECT_ID(N'dbo.tr_Posts_Audit_No_Safeties', N'TR') IS NOT NULL
BEGIN
 DROP TRIGGER dbo.tr_Posts_Audit_No_Safeties;
END;
GO

CREATE TRIGGER
 dbo.tr_Posts_Audit_No_Safeties
ON dbo.Posts
AFTER INSERT, UPDATE, DELETE
AS
BEGIN
 SET NOCOUNT ON;
 /* Update LastActivityDate on INSERT and UPDATE */
 UPDATE
 p
 SET
 p.LastActivityDate = GETDATE()
 FROM dbo.Posts AS p
 JOIN inserted AS i
 ON i.Id = p.Id;
 /* Handle INSERT - only when deleted table is empty */
 INSERT INTO
 dbo.AuditPosts
 (
 PostsId, Action, ActivityDate, ColumnName, ColumnValue
 )
 SELECT
 PostsId = i.Id,
 Action = 'I',
 ActivityDate = GETDATE(),
 ColumnName = N'Title',
 ColumnValue = CONVERT(nvarchar(250), i.Title)
 FROM inserted AS i
 WHERE NOT EXISTS
 (
 SELECT
 1/0
 FROM deleted AS d
 )
 AND i.Title IS NOT NULL

 UNION ALL

 SELECT
 PostsId = i.Id,
 Action = 'I',
 ActivityDate = GETDATE(),
 ColumnName = N'Title',
 ColumnValue = NULL
 FROM inserted AS i
 WHERE NOT EXISTS
 (
 SELECT
 1/0
 FROM deleted AS d
 )
 AND i.Title IS NULL

 UNION ALL

 SELECT
 PostsId = i.Id,
 Action = 'I',
 ActivityDate = GETDATE(),
 ColumnName = N'PostTypeId',
 ColumnValue = CONVERT(nvarchar(250), i.PostTypeId)
 FROM inserted AS i
 WHERE NOT EXISTS
 (
 SELECT
 1/0
 FROM deleted AS d
 )

 UNION ALL

 SELECT
 PostsId = i.Id,
 Action = 'I',
 ActivityDate = GETDATE(),
 ColumnName = N'OwnerUserId',
 ColumnValue = CONVERT(nvarchar(250), i.OwnerUserId)
 FROM inserted AS i
 WHERE NOT EXISTS
 (
 SELECT
 1/0
 FROM deleted AS d
 )
 AND i.OwnerUserId IS NOT NULL

 UNION ALL

 SELECT
 PostsId = i.Id,
 Action = 'I',
 ActivityDate = GETDATE(),
 ColumnName = N'OwnerUserId',
 ColumnValue = NULL
 FROM inserted AS i
 WHERE NOT EXISTS
 (
 SELECT
 1/0
 FROM deleted AS d
 )
 AND i.OwnerUserId IS NULL

 UNION ALL

 SELECT
 PostsId = i.Id,
 Action = 'I',
 ActivityDate = GETDATE(),
 ColumnName = N'Score',
 ColumnValue = CONVERT(nvarchar(250), i.Score)
 FROM inserted AS i
 WHERE NOT EXISTS
 (
 SELECT
 1/0
 FROM deleted AS d
 );
 /* Handle UPDATE - only if Title, PostTypeId, OwnerUserId, or Score changed */
 INSERT INTO
 dbo.AuditPosts
 (
 PostsId, Action, ActivityDate, ColumnName, ColumnValue
 )
 SELECT
 PostsId = d.Id,
 Action = 'U',
 ActivityDate = GETDATE(),
 ColumnName = N'Title',
 ColumnValue = CONVERT(nvarchar(250), d.Title)
 FROM inserted AS i
 JOIN deleted AS d
 ON d.Id = i.Id
 WHERE
 (
 ISNULL(i.Title, N'') <> ISNULL(d.Title, N'')
 OR i.PostTypeId <> d.PostTypeId
 OR ISNULL(i.OwnerUserId, 0) <> ISNULL(d.OwnerUserId, 0)
 OR i.Score <> d.Score
 )
 AND d.Title IS NOT NULL

 UNION ALL

 SELECT
 PostsId = d.Id,
 Action = 'U',
 ActivityDate = GETDATE(),
 ColumnName = N'Title',
 ColumnValue = NULL
 FROM inserted AS i
 JOIN deleted AS d
 ON d.Id = i.Id
 WHERE
 (
 ISNULL(i.Title, N'') <> ISNULL(d.Title, N'')
 OR i.PostTypeId <> d.PostTypeId
 OR ISNULL(i.OwnerUserId, 0) <> ISNULL(d.OwnerUserId, 0)
 OR i.Score <> d.Score
 )
 AND d.Title IS NULL

 UNION ALL

 SELECT
 PostsId = d.Id,
 Action = 'U',
 ActivityDate = GETDATE(),
 ColumnName = N'PostTypeId',
 ColumnValue = CONVERT(nvarchar(250), d.PostTypeId)
 FROM inserted AS i
 JOIN deleted AS d
 ON d.Id = i.Id
 WHERE
 (
 ISNULL(i.Title, N'') <> ISNULL(d.Title, N'')
 OR i.PostTypeId <> d.PostTypeId
 OR ISNULL(i.OwnerUserId, 0) <> ISNULL(d.OwnerUserId, 0)
 OR i.Score <> d.Score
 )

 UNION ALL

 SELECT
 PostsId = d.Id,
 Action = 'U',
 ActivityDate = GETDATE(),
 ColumnName = N'OwnerUserId',
 ColumnValue = CONVERT(nvarchar(250), d.OwnerUserId)
 FROM inserted AS i
 JOIN deleted AS d
 ON d.Id = i.Id
 WHERE
 (
 ISNULL(i.Title, N'') <> ISNULL(d.Title, N'')
 OR i.PostTypeId <> d.PostTypeId
 OR ISNULL(i.OwnerUserId, 0) <> ISNULL(d.OwnerUserId, 0)
 OR i.Score <> d.Score
 )
 AND d.OwnerUserId IS NOT NULL

 UNION ALL

 SELECT
 PostsId = d.Id,
 Action = 'U',
 ActivityDate = GETDATE(),
 ColumnName = N'OwnerUserId',
 ColumnValue = NULL
 FROM inserted AS i
 JOIN deleted AS d
 ON d.Id = i.Id
 WHERE
 (
 ISNULL(i.Title, N'') <> ISNULL(d.Title, N'')
 OR i.PostTypeId <> d.PostTypeId
 OR ISNULL(i.OwnerUserId, 0) <> ISNULL(d.OwnerUserId, 0)
 OR i.Score <> d.Score
 )
 AND d.OwnerUserId IS NULL

 UNION ALL

 SELECT
 PostsId = d.Id,
 Action = 'U',
 ActivityDate = GETDATE(),
 ColumnName = N'Score',
 ColumnValue = CONVERT(nvarchar(250), d.Score)
 FROM inserted AS i
 JOIN deleted AS d
 ON d.Id = i.Id
 WHERE
 (
 ISNULL(i.Title, N'') <> ISNULL(d.Title, N'')
 OR i.PostTypeId <> d.PostTypeId
 OR ISNULL(i.OwnerUserId, 0) <> ISNULL(d.OwnerUserId, 0)
 OR i.Score <> d.Score
 );
 /* Handle DELETE - use LEFT JOIN to inserted and check if inserted.Id IS NULL */
 INSERT INTO
 dbo.AuditPosts
 (
 PostsId, Action, ActivityDate, ColumnName, ColumnValue
 )
 SELECT
 PostsId = d.Id,
 Action = 'D',
 ActivityDate = GETDATE(),
 ColumnName = N'Title',
 ColumnValue = CONVERT(nvarchar(250), d.Title)
 FROM deleted AS d
 LEFT JOIN inserted AS i
 ON i.Id = d.Id
 WHERE i.Id IS NULL
 AND d.Title IS NOT NULL

 UNION ALL

 SELECT
 PostsId = d.Id,
 Action = 'D',
 ActivityDate = GETDATE(),
 ColumnName = N'Title',
 ColumnValue = NULL
 FROM deleted AS d
 LEFT JOIN inserted AS i
 ON i.Id = d.Id
 WHERE i.Id IS NULL
 AND d.Title IS NULL

 UNION ALL

 SELECT
 PostsId = d.Id,
 Action = 'D',
 ActivityDate = GETDATE(),
 ColumnName = N'PostTypeId',
 ColumnValue = CONVERT(nvarchar(250), d.PostTypeId)
 FROM deleted AS d
 LEFT JOIN inserted AS i
 ON i.Id = d.Id
 WHERE i.Id IS NULL

 UNION ALL

 SELECT
 PostsId = d.Id,
 Action = 'D',
 ActivityDate = GETDATE(),
 ColumnName = N'OwnerUserId',
 ColumnValue = CONVERT(nvarchar(250), d.OwnerUserId)
 FROM deleted AS d
 LEFT JOIN inserted AS i
 ON i.Id = d.Id
 WHERE i.Id IS NULL
 AND d.OwnerUserId IS NOT NULL

 UNION ALL

 SELECT
 PostsId = d.Id,
 Action = 'D',
 ActivityDate = GETDATE(),
 ColumnName = N'OwnerUserId',
 ColumnValue = NULL
 FROM deleted AS d
 LEFT JOIN inserted AS i
 ON i.Id = d.Id
 WHERE i.Id IS NULL
 AND d.OwnerUserId IS NULL

 UNION ALL

 SELECT
 PostsId = d.Id,
 Action = 'D',
 ActivityDate = GETDATE(),
 ColumnName = N'Score',
 ColumnValue = CONVERT(nvarchar(250), d.Score)
 FROM deleted AS d
 LEFT JOIN inserted AS i
 ON i.Id = d.Id
 WHERE i.Id IS NULL;
END;
GO

Setup: Create Query Store History Capture Helper

To track which Query Store query IDs correspond to which operations, we’ll create a helper procedure that captures Query Store data after each step in the test procedure. This eliminates guesswork when analyzing Query Store output.

The helper procedure queries Query Store for all trigger-related queries and saves them to a history table with a step description. This lets us see exactly which query IDs were created or updated after each operation.

/* Create table to store Query Store history with step descriptions */
IF OBJECT_ID(N'dbo.QueryStoreHistory', N'U') IS NOT NULL
BEGIN
 DROP TABLE dbo.QueryStoreHistory;
END;
GO

CREATE TABLE
 dbo.QueryStoreHistory
(
 history_id bigint NOT NULL IDENTITY(1,1) PRIMARY KEY,
 step_number integer NOT NULL,
 step_description nvarchar(100) NOT NULL,
 query_id bigint NOT NULL,
 query_text_id bigint NOT NULL,
 plan_id bigint NOT NULL,
 runtime_stats_id bigint NOT NULL,
 context_settings_id bigint NOT NULL,
 set_options_decoded nvarchar(max) NOT NULL,
 query_sql_text nvarchar(max) NOT NULL,
 count_executions bigint NOT NULL,
 count_compiles bigint NOT NULL,
 avg_cpu_time decimal(38,2) NOT NULL,
 avg_duration decimal(38,2) NOT NULL,
 avg_logical_io_reads decimal(38,2) NOT NULL,
 last_rowcount bigint NOT NULL,
 min_rowcount bigint NOT NULL,
 max_rowcount bigint NOT NULL,
 stdev_rowcount decimal(38,2) NOT NULL,
 last_execution_time datetimeoffset(7) NOT NULL,
 captured_at datetime2(7) NOT NULL DEFAULT SYSDATETIME()
);
GO

/* Helper procedure to capture Query Store data for trigger queries */
CREATE OR ALTER PROCEDURE
 dbo.sp_CaptureQueryStoreHistory
(
 @step_number integer, /* Step number for ordering (1, 2, 3, etc.) */
 @step_description nvarchar(100) /* Description of what step just executed */
)
AS
BEGIN
 SET NOCOUNT ON;

 /* Strip HTML tags from step description before storing */
 DECLARE
 @clean_description nvarchar(100) = @step_description;

 /* Remove common HTML tags */
 SET @clean_description = REPLACE(@clean_description, N'<strong>', N'');
 SET @clean_description = REPLACE(@clean_description, N'</strong>', N'');

 INSERT INTO
 dbo.QueryStoreHistory
 (
 step_number,
 step_description,
 query_id,
 query_text_id,
 plan_id,
 runtime_stats_id,
 context_settings_id,
 set_options_decoded,
 query_sql_text,
 count_executions,
 count_compiles,
 avg_cpu_time,
 avg_duration,
 avg_logical_io_reads,
 last_rowcount,
 min_rowcount,
 max_rowcount,
 stdev_rowcount,
 last_execution_time
 )
 SELECT
 step_number = @step_number,
 step_description = @clean_description,
 query_id = latest_stats.query_id,
 query_text_id = ISNULL(latest_stats.query_text_id, 0),
 plan_id = ISNULL(latest_stats.plan_id, 0),
 runtime_stats_id = ISNULL(latest_stats.runtime_stats_id, 0),
 context_settings_id = ISNULL(latest_stats.context_settings_id, 0),
 set_options_decoded = latest_stats.set_options_decoded,
 query_sql_text = ISNULL(latest_stats.query_sql_text, N''),
 count_executions = ISNULL(latest_stats.count_executions, 0),
 count_compiles = ISNULL(latest_stats.count_compiles, 0),
 avg_cpu_time = ISNULL(latest_stats.avg_cpu_time, 0),
 avg_duration = ISNULL(latest_stats.avg_duration, 0),
 avg_logical_io_reads = ISNULL(latest_stats.avg_logical_io_reads, 0),
 last_rowcount = ISNULL(latest_stats.last_rowcount, 0),
 min_rowcount = ISNULL(latest_stats.min_rowcount, 0),
 max_rowcount = ISNULL(latest_stats.max_rowcount, 0),
 stdev_rowcount = ISNULL(latest_stats.stdev_rowcount, 0),
 last_execution_time = ISNULL(latest_stats.last_execution_time, CAST('1900-01-01' AS datetimeoffset(7)))
 FROM
 (
 /* Get latest runtime stats entry for each plan */
 SELECT
 qsq.query_id,
 qsq.query_text_id,
 qsp.plan_id,
 qsrs.runtime_stats_id,
 qsq.context_settings_id,
 set_options_decoded =
 ISNULL
 (
 STUFF
 (
 CASE WHEN (qcs.set_options & 1) = 1 THEN ', ANSI_PADDING ON' ELSE ', ANSI_PADDING OFF' END +
 CASE WHEN (qcs.set_options & 2) = 2 THEN ', ANSI_NULLS ON' ELSE ', ANSI_NULLS OFF' END +
 CASE WHEN (qcs.set_options & 4) = 4 THEN ', QUOTED_IDENTIFIER ON' ELSE ', QUOTED_IDENTIFIER OFF' END +
 CASE WHEN (qcs.set_options & 8) = 8 THEN ', ARITHABORT ON' ELSE ', ARITHABORT OFF' END +
 CASE WHEN (qcs.set_options & 16) = 16 THEN ', ARITHIGNORE ON' ELSE ', ARITHIGNORE OFF' END +
 CASE WHEN (qcs.set_options & 32) = 32 THEN ', NUMERIC_ROUNDABORT ON' ELSE ', NUMERIC_ROUNDABORT OFF' END +
 CASE WHEN (qcs.set_options & 64) = 64 THEN ', ANSI_WARNINGS ON' ELSE ', ANSI_WARNINGS OFF' END +
 CASE WHEN (qcs.set_options & 128) = 128 THEN ', ANSI_NULL_DFLT_ON ON' ELSE ', ANSI_NULL_DFLT_ON OFF' END +
 CASE WHEN (qcs.set_options & 256) = 256 THEN ', CONCAT_NULL_YIELDS_NULL ON' ELSE ', CONCAT_NULL_YIELDS_NULL OFF' END +
 CASE WHEN (qcs.set_options & 512) = 512 THEN ', CURSOR_CLOSE_ON_COMMIT ON' ELSE ', CURSOR_CLOSE_ON_COMMIT OFF' END +
 CASE WHEN (qcs.set_options & 1024) = 1024 THEN ', IMPLICIT_TRANSACTIONS ON' ELSE ', IMPLICIT_TRANSACTIONS OFF' END +
 CASE WHEN (qcs.set_options & 2048) = 2048 THEN ', TEXTSIZE 2147483647' ELSE ', TEXTSIZE default' END +
 CASE WHEN (qcs.set_options & 4096) = 4096 THEN ', DATE FIRST 1' ELSE ', DATE FIRST 7' END +
 CASE WHEN (qcs.set_options & 8192) = 8192 THEN ', DATE FORMAT mdy' ELSE ', DATE FORMAT dmy' END +
 CASE WHEN (qcs.set_options & 16384) = 16384 THEN ', LANGUAGE us_english' ELSE ', LANGUAGE other' END,
 1, 2, ''
 ),
 N'UNKNOWN'
 ),
 query_sql_text = qst.query_sql_text,
 qsrs.count_executions,
 qsq.count_compiles,
 qsrs.avg_cpu_time,
 qsrs.avg_duration,
 qsrs.avg_logical_io_reads,
 qsrs.last_rowcount,
 qsrs.min_rowcount,
 qsrs.max_rowcount,
 qsrs.stdev_rowcount,
 qsrs.last_execution_time,
 row_number =
 ROW_NUMBER() OVER
 (
 PARTITION BY
 qsp.plan_id
 ORDER BY
 qsrs.runtime_stats_id DESC
 )
 FROM sys.query_store_query AS qsq
 JOIN sys.query_store_query_text AS qst
 ON qsq.query_text_id = qst.query_text_id
 JOIN sys.query_store_plan AS qsp
 ON qsp.query_id = qsq.query_id
 JOIN sys.query_store_runtime_stats AS qsrs
 ON qsrs.plan_id = qsp.plan_id
 LEFT JOIN sys.query_context_settings AS qcs
 ON qcs.context_settings_id = qsq.context_settings_id
 WHERE qsq.object_id = OBJECT_ID(N'dbo.tr_Posts_Audit_No_Safeties', N'TR')
 ) AS latest_stats
 WHERE latest_stats.row_number = 1;
END;
GO

🔥 Note: The set_options_decoded column decodes the set_options bitmask into human-readable SET option values. The bitmask decoding logic uses standard SQL Server SET option bitmask values documented in Microsoft's documentation for sys.dm_exec_plan_attributes and sys.query_context_settings. For more information on how SET options affect query performance and execution plans, see Erland Sommarskog's article "Slow in the Application, Fast in SSMS?".

The sp_CaptureQueryStoreHistory procedure queries Query Store for all queries related to the trigger and saves detailed information to dbo.QueryStoreHistory with a step description. It captures all rows from Query Store without grouping, including:

Query ID, query text ID, plan ID, and runtime stats ID
Context settings ID and decoded SET options (including IMPLICIT_TRANSACTIONS status)
Query SQL text
Runtime statistics (execution count, CPU time, duration, logical I/O reads, last execution time)

The test procedure calls this helper after each step, building a detailed history that maps Query Store data to specific operations. Analyze and aggregate this data later as needed.

Setup: Create Test Data Procedure

Now let’s create a stored procedure that fires the trigger in various ways. The procedure also includes a MERGE statement and dynamic SQL with sp_executesql (inspired by Paul White’s answer on StackOverflow) to demonstrate additional trigger firing scenarios:

CREATE OR ALTER PROCEDURE
 dbo.sp_TestTriggerFires
AS
BEGIN
 /* Set consistent options for Query Store tracking */
 SET NOCOUNT ON;
 SET ANSI_NULLS ON;
 SET QUOTED_IDENTIFIER ON;
 SET ARITHABORT ON;
 SET NUMERIC_ROUNDABORT OFF;
 SET IMPLICIT_TRANSACTIONS OFF;

 /* Store the IDs for later updates and deletes */
 DECLARE
 @inserted_ids TABLE
 (
 Id integer NOT NULL PRIMARY KEY,
 PostTypeId integer NOT NULL
 );

 DECLARE
 @question_ids TABLE
 (
 Id integer NOT NULL PRIMARY KEY
 );

 DECLARE
 @answer_ids TABLE
 (
 Id integer NOT NULL PRIMARY KEY
 );

 /* <strong>Step A:</strong> Insert 10 rows: 5 questions (PostTypeId=1), 5 answers (PostTypeId=2) */
 /* Use OUTPUT clause to capture IDs immediately - no table scan needed */
 INSERT INTO
 dbo.Posts
 (
 AcceptedAnswerId, AnswerCount, Body, CommentCount, CreationDate,
 FavoriteCount, LastActivityDate, LastEditDate, OwnerUserId, ParentId,
 PostTypeId, Score, Tags, Title, ViewCount
 )
 OUTPUT
 inserted.Id, inserted.PostTypeId
 INTO
 @inserted_ids (Id, PostTypeId)
 VALUES
 (NULL, 0, N'Test question body for trigger testing', 0, GETDATE(), 0, GETDATE(), NULL, 1, NULL, 1, 0, N'<test>', N'Test Question 1', 0),
 (NULL, 0, N'Test question body 2', 0, GETDATE(), 0, GETDATE(), NULL, 2, NULL, 1, 0, N'<test>', N'Test Question 2', 0),
 (NULL, 0, N'Test question body 3', 0, GETDATE(), 0, GETDATE(), NULL, 3, NULL, 1, 0, N'<test>', N'Test Question 3', 0),
 (NULL, 0, N'Test question body 4', 0, GETDATE(), 0, GETDATE(), NULL, 4, NULL, 1, 0, N'<test>', N'Test Question 4', 0),
 (NULL, 0, N'Test question body 5', 0, GETDATE(), 0, GETDATE(), NULL, 5, NULL, 1, 0, N'<test>', N'Test Question 5', 0),
 (NULL, 0, N'Test answer body 1', 0, GETDATE(), 0, GETDATE(), NULL, 6, 1, 2, 0, NULL, NULL, 0),
 (NULL, 0, N'Test answer body 2', 0, GETDATE(), 0, GETDATE(), NULL, 7, 1, 2, 0, NULL, NULL, 0),
 (NULL, 0, N'Test answer body 3', 0, GETDATE(), 0, GETDATE(), NULL, 8, 1, 2, 0, NULL, NULL, 0),
 (NULL, 0, N'Test answer body 4', 0, GETDATE(), 0, GETDATE(), NULL, 9, 1, 2, 0, NULL, NULL, 0),
 (NULL, 0, N'Test answer body 5', 0, GETDATE(), 0, GETDATE(), NULL, 10, 1, 2, 0, NULL, NULL, 0);

 /* Split inserted IDs into question and answer tables based on PostTypeId */
 INSERT INTO
 @question_ids
 (
 Id
 )
 SELECT
 ii.Id
 FROM @inserted_ids AS ii
 WHERE ii.PostTypeId = 1;

 INSERT INTO
 @answer_ids
 (
 Id
 )
 SELECT
 ii.Id
 FROM @inserted_ids AS ii
 WHERE ii.PostTypeId = 2;

 /* Capture Query Store after Step A */
 EXECUTE dbo.sp_CaptureQueryStoreHistory
 @step_number = 1,
 @step_description = N'<strong>Step A:</strong> INSERT 10 rows';

 /* Update all 10 rows with various changes */
 /* Question 1: Update Title */
 UPDATE
 p
 SET
 p.Title = N'Updated Test Question 1'
 FROM dbo.Posts AS p
 JOIN @question_ids AS q
 ON q.Id = p.Id
 WHERE p.Id =
 (
 SELECT TOP (1)
 q2.Id
 FROM @question_ids AS q2
 ORDER BY
 q2.Id
 );

 /* Capture Query Store after Step B */
 EXECUTE dbo.sp_CaptureQueryStoreHistory
 @step_number = 2,
 @step_description = N'<strong>Step B:</strong> UPDATE Question 1 (Title)';

 /* Question 2: Update CommentCount (should NOT trigger audit) */
 UPDATE
 p
 SET
 p.CommentCount = 5
 FROM dbo.Posts AS p
 JOIN @question_ids AS q
 ON q.Id = p.Id
 WHERE p.Id =
 (
 SELECT
 q2.Id
 FROM @question_ids AS q2
 ORDER BY
 q2.Id
 OFFSET 1 ROWS
 FETCH NEXT 1 ROW ONLY
 );

 /* Capture Query Store after Step C */
 EXECUTE dbo.sp_CaptureQueryStoreHistory
 @step_number = 3,
 @step_description = N'<strong>Step C:</strong> UPDATE Question 2 (CommentCount)';

 /* Question 3: Update Body (should NOT trigger audit) */
 UPDATE
 p
 SET
 p.Body = N'Updated body text'
 FROM dbo.Posts AS p
 JOIN @question_ids AS q
 ON q.Id = p.Id
 WHERE p.Id =
 (
 SELECT
 q2.Id
 FROM @question_ids AS q2
 ORDER BY
 q2.Id
 OFFSET 2 ROWS
 FETCH NEXT 1 ROW ONLY
 );

 /* Capture Query Store after Step D */
 EXECUTE dbo.sp_CaptureQueryStoreHistory
 @step_number = 4,
 @step_description = N'<strong>Step D:</strong> UPDATE Question 3 (Body)';

 /* Question 4: Update LastEditDate (should NOT trigger audit) */
 UPDATE
 p
 SET
 p.LastEditDate = GETDATE()
 FROM dbo.Posts AS p
 JOIN @question_ids AS q
 ON q.Id = p.Id
 WHERE p.Id =
 (
 SELECT
 q2.Id
 FROM @question_ids AS q2
 ORDER BY
 q2.Id
 OFFSET 3 ROWS
 FETCH NEXT 1 ROW ONLY
 );

 /* Capture Query Store after Step E */
 EXECUTE dbo.sp_CaptureQueryStoreHistory
 @step_number = 5,
 @step_description = N'<strong>Step E:</strong> UPDATE Question 4 (LastEditDate)';

 /* Question 5: Update Score */
 UPDATE
 p
 SET
 p.Score = 10
 FROM dbo.Posts AS p
 JOIN @question_ids AS q
 ON q.Id = p.Id
 WHERE p.Id =
 (
 SELECT
 q2.Id
 FROM @question_ids AS q2
 ORDER BY
 q2.Id
 OFFSET 4 ROWS
 FETCH NEXT 1 ROW ONLY
 );

 /* Capture Query Store after Step F */
 EXECUTE dbo.sp_CaptureQueryStoreHistory
 @step_number = 6,
 @step_description = N'<strong>Step F:</strong> UPDATE Question 5 (Score)';

 /* Answer 1: Update OwnerUserId */
 UPDATE
 p
 SET
 p.OwnerUserId = 100
 FROM dbo.Posts AS p
 JOIN @answer_ids AS a
 ON a.Id = p.Id
 WHERE p.Id =
 (
 SELECT TOP (1)
 a2.Id
 FROM @answer_ids AS a2
 ORDER BY
 a2.Id
 );

 /* Capture Query Store after Step G */
 EXECUTE dbo.sp_CaptureQueryStoreHistory
 @step_number = 7,
 @step_description = N'<strong>Step G:</strong> UPDATE Answer 1 (OwnerUserId)';

 /* Answer 2: Update CommentCount (should NOT trigger audit) */
 UPDATE
 p
 SET
 p.CommentCount = 3
 FROM dbo.Posts AS p
 JOIN @answer_ids AS a
 ON a.Id = p.Id
 WHERE p.Id =
 (
 SELECT
 a2.Id
 FROM @answer_ids AS a2
 ORDER BY
 a2.Id
 OFFSET 1 ROWS
 FETCH NEXT 1 ROW ONLY
 );

 /* Capture Query Store after Step H */
 EXECUTE dbo.sp_CaptureQueryStoreHistory
 @step_number = 8,
 @step_description = N'<strong>Step H:</strong> UPDATE Answer 2 (CommentCount)';

 /* Answer 3: Update Body (should NOT trigger audit) */
 UPDATE
 p
 SET
 p.Body = N'Updated answer body'
 FROM dbo.Posts AS p
 JOIN @answer_ids AS a
 ON a.Id = p.Id
 WHERE p.Id =
 (
 SELECT
 a2.Id
 FROM @answer_ids AS a2
 ORDER BY
 a2.Id
 OFFSET 2 ROWS
 FETCH NEXT 1 ROW ONLY
 );

 /* Capture Query Store after Step I */
 EXECUTE dbo.sp_CaptureQueryStoreHistory
 @step_number = 9,
 @step_description = N'<strong>Step I:</strong> UPDATE Answer 3 (Body)';

 /* Answer 4: Update LastEditDate (should NOT trigger audit) */
 UPDATE
 p
 SET
 p.LastEditDate = GETDATE()
 FROM dbo.Posts AS p
 JOIN @answer_ids AS a
 ON a.Id = p.Id
 WHERE p.Id =
 (
 SELECT
 a2.Id
 FROM @answer_ids AS a2
 ORDER BY
 a2.Id
 OFFSET 3 ROWS
 FETCH NEXT 1 ROW ONLY
 );

 /* Capture Query Store after Step J */
 EXECUTE dbo.sp_CaptureQueryStoreHistory
 @step_number = 10,
 @step_description = N'<strong>Step J:</strong> UPDATE Answer 4 (LastEditDate)';

 /* Answer 5: Update Score */
 UPDATE
 p
 SET
 p.Score = 5
 FROM dbo.Posts AS p
 JOIN @answer_ids AS a
 ON a.Id = p.Id
 WHERE p.Id =
 (
 SELECT
 a2.Id
 FROM @answer_ids AS a2
 ORDER BY
 a2.Id
 OFFSET 4 ROWS
 FETCH NEXT 1 ROW ONLY
 );

 /* Capture Query Store after Step K */
 EXECUTE dbo.sp_CaptureQueryStoreHistory
 @step_number = 11,
 @step_description = N'<strong>Step K:</strong> UPDATE Answer 5 (Score)';

 /* Delete all 10 rows */
 DELETE
 p
 FROM dbo.Posts AS p
 WHERE p.Id IN
 (
 SELECT
 q.Id
 FROM @question_ids AS q

 UNION ALL

 SELECT
 a.Id
 FROM @answer_ids AS a
 );

 /* Capture Query Store after Step L */
 EXECUTE dbo.sp_CaptureQueryStoreHistory
 @step_number = 12,
 @step_description = N'<strong>Step L:</strong> DELETE 10 rows';

 /* MERGE statement */
 /* If a matching row exists, update it; otherwise insert a new row */
 /* This fires the trigger on Posts, which writes to AuditPosts */
 DECLARE
 @merge_matched_count integer = 0,
 @merge_not_matched_count integer = 0;

 CREATE TABLE
 #merge_output
 (
 action nvarchar(10) NOT NULL,
 id integer NOT NULL
 );

 MERGE
 dbo.Posts AS target
 USING
 (
 SELECT TOP (1)
 Id = p.Id,
 Title = N'MERGE: Where Reality Never Lives Up To Expectations'
 FROM dbo.Posts AS p
 WHERE p.CreationDate >= DATEADD(MINUTE, -1, GETDATE())
 ORDER BY
 p.CreationDate DESC

 UNION ALL

 SELECT
 Id = -1,
 Title = N'MERGE: Where Reality Never Lives Up To Expectations'
 WHERE NOT EXISTS
 (
 SELECT
 1/0
 FROM dbo.Posts AS p2
 WHERE p2.CreationDate >= DATEADD(MINUTE, -1, GETDATE())
 )
 ) AS source
 ON target.Id = source.Id
 WHEN MATCHED
 THEN
 UPDATE
 SET
 target.LastActivityDate = GETDATE()
 WHEN NOT MATCHED BY TARGET
 THEN
 INSERT
 (
 AcceptedAnswerId, AnswerCount, Body, CommentCount, CreationDate,
 FavoriteCount, LastActivityDate, LastEditDate, OwnerUserId, ParentId,
 PostTypeId, Score, Tags, Title, ViewCount
 )
 VALUES
 (
 NULL, 0, N'Test row inserted via MERGE', 0, GETDATE(), 0, GETDATE(), NULL, 1, NULL, 1, 0, N'<test>', N'Merged Row', 0
 )
 OUTPUT
 $action,
 inserted.Id
 INTO
 #merge_output (action, id);

 SELECT
 @merge_matched_count = COUNT_BIG(*)
 FROM #merge_output
 WHERE action = N'UPDATE';

 SELECT
 @merge_not_matched_count = COUNT_BIG(*)
 FROM #merge_output
 WHERE action = N'INSERT';

 PRINT N'Step M (MERGE): ' +
 CONVERT(nvarchar(10), @merge_matched_count) + N' row(s) updated (WHEN MATCHED), ' +
 CONVERT(nvarchar(10), @merge_not_matched_count) + N' row(s) inserted (WHEN NOT MATCHED BY TARGET).';

 /* Capture Query Store after Step M */
 EXECUTE dbo.sp_CaptureQueryStoreHistory
 @step_number = 13,
 @step_description = N'<strong>Step M:</strong> MERGE statement';

 /* Dynamic SQL with sp_executesql */
 /* This fires the trigger on Posts via DELETE, which writes to AuditPosts */
 /* Delete the row that Step M (MERGE statement) modified - MERGE always creates or updates a row */
 /* Find the row with the most recent LastActivityDate that was updated in the last minute */
 DECLARE
 @sql nvarchar(max) = N'',
 @posts_id integer = 0;

 SELECT
 @posts_id = MAX(p.Id)
 FROM dbo.Posts AS p
 WHERE p.LastActivityDate >= DATEADD(MINUTE, -1, GETDATE())
 AND p.LastActivityDate =
 (
 SELECT
 MAX(p2.LastActivityDate)
 FROM dbo.Posts AS p2
 WHERE p2.LastActivityDate >= DATEADD(MINUTE, -1, GETDATE())
 );

 IF @posts_id IS NULL OR @posts_id <= 0
 BEGIN
 RAISERROR(N'Dynamic SQL DELETE (Step N) did not find a row to delete. Expected at least one row with LastActivityDate updated by Step M (MERGE statement) in the last minute.', 16, 1);
 RETURN;
 END;

 SET @sql = N'
 DELETE /* run via dynamic sql */
 p
 FROM dbo.Posts AS p
 WHERE p.Id = @posts_id;
 ';

 EXECUTE sys.sp_executesql
 @sql,
 N'@posts_id integer',
 @posts_id;

 /* Capture Query Store after Step N */
 EXECUTE dbo.sp_CaptureQueryStoreHistory
 @step_number = 14,
 @step_description = N'<strong>Step N:</strong> Dynamic SQL DELETE';
END;
GO

The test procedure performs the following operations in order:

Step	Operation	Details	Notes
A	INSERT 10 rows	Single INSERT statement with VALUES clause inserting 5 questions (PostTypeId=1) and 5 answers (PostTypeId=2)	Updates LastActivityDate (10 times), adds 4 rows to AuditPosts per row (Action=‘I’)
B	UPDATE Question 1	Updates Title column	Updates LastActivityDate, adds 4 rows to AuditPosts (Action=‘U’ - Title is monitored)
C	UPDATE Question 2	Updates CommentCount column	Updates LastActivityDate only (CommentCount not monitored)
D	UPDATE Question 3	Updates Body column	Updates LastActivityDate only (Body not monitored)
E	UPDATE Question 4	Updates LastEditDate column	Updates LastActivityDate only (LastEditDate not monitored)
F	UPDATE Question 5	Updates Score column	Updates LastActivityDate, adds 4 rows to AuditPosts (Action=‘U’ - Score is monitored)
G	UPDATE Answer 1	Updates OwnerUserId column	Updates LastActivityDate, adds 4 rows to AuditPosts (Action=‘U’ - OwnerUserId is monitored)
H	UPDATE Answer 2	Updates CommentCount column	Updates LastActivityDate only (CommentCount not monitored)
I	UPDATE Answer 3	Updates Body column	Updates LastActivityDate only (Body not monitored)
J	UPDATE Answer 4	Updates LastEditDate column	Updates LastActivityDate only (LastEditDate not monitored)
K	UPDATE Answer 5	Updates Score column	Updates LastActivityDate, adds 4 rows to AuditPosts (Action=‘U’ - Score is monitored)
L	DELETE 10 rows	Single DELETE statement removing all 10 inserted rows	Adds 4 rows to AuditPosts per row (Action=‘D’), does not update LastActivityDate
M	MERGE statement	MERGE operation that updates LastActivityDate if row exists, or inserts new row if not	Updates LastActivityDate, adds 4 rows to AuditPosts (Action=‘I’ in this case - inserts new row)
N	Dynamic SQL DELETE	DELETE statement executed via `sp_executesql`	Adds 4 rows to AuditPosts (Action=‘D’), does not update LastActivityDate. Deletes row from Step M.

Summary:

Steps overview: A (INSERT 10 rows), B-K (10 individual UPDATEs), L (DELETE 10 rows), M (MERGE statement), N (Dynamic SQL DELETE)
Trigger fires: 14 total (1 INSERT statement, 10 UPDATE statements, 1 DELETE statement, 1 MERGE statement, 1 dynamic SQL DELETE statement). The trigger fires once per DML statement, not once per row.
LastActivityDate updates: 21 total rows updated (10 rows from step A INSERT, 10 rows from steps B-K UPDATEs, 1 row from step M MERGE)
AuditPosts inserts: 104 total rows (44 Action=‘I’ from steps A and M, 16 Action=‘U’ from steps B/F/G/K, 44 Action=‘D’ from steps L and N)
Operations that update LastActivityDate: All INSERT and UPDATE operations (steps A-K, M)
Operations that don’t update LastActivityDate: All DELETE operations (steps L, N) because DELETE operations don’t have an inserted table

Run Test Procedure and Review Audit Table Rows Inserted by Trigger

Let’s run the procedure and check the audit table:

SET ANSI_NULLS ON;
SET QUOTED_IDENTIFIER ON;
SET ARITHABORT ON;
SET NUMERIC_ROUNDABORT OFF;
SET IMPLICIT_TRANSACTIONS OFF;
GO

EXECUTE dbo.sp_TestTriggerFires;
GO

/* Review audit results */
SELECT
 ap.Action,
 audit_count = COUNT_BIG(*),
 posts_affected = COUNT_BIG(DISTINCT ap.PostsId)
FROM dbo.AuditPosts AS ap
GROUP BY
 ap.Action
ORDER BY
 ap.Action;
GO

/* Detailed view */
SELECT
 ap.Action,
 ap.PostsId,
 ap.ColumnName,
 ap.ColumnValue,
 ap.ActivityDate
FROM dbo.AuditPosts AS ap
ORDER BY
 ap.PostsId,
 ap.Action,
 ap.ColumnName;
GO

Each run of dbo.sp_TestTriggerFires should add 104 rows to dbo.AuditPosts:

44 rows with Action = ‘I’ (11 inserts × 4 columns each: 10 from Step A, 1 from Step M MERGE when it inserts)
16 rows with Action = ‘U’ (4 updates × 4 columns each: from Steps B, F, G, K that change monitored columns)
44 rows with Action = ‘D’ (11 deletes × 4 columns each: 10 from Step L, 1 from Step N dynamic SQL DELETE)

Review Query Store Data

Since we captured Query Store data after each step in the dbo.QueryStoreHistory table, we can analyze what changed between steps.

/* Summary: final state for each unique query_id and context_settings_id combination */
SELECT
 latest.query_id,
 latest.context_settings_id,
 /* Only show IMPLICIT_TRANSACTIONS since that's the key difference */
 implicit_transactions =
 CASE
 WHEN latest.set_options_decoded LIKE N'%IMPLICIT_TRANSACTIONS ON%'
 THEN N'ON'
 ELSE N'OFF'
 END,
 /* Summarize query text to identify trigger operations */
 query_summary =
 CASE
 WHEN latest.query_sql_text LIKE N'%UPDATE%LastActivityDate%'
 THEN N'UPDATE LastActivityDate'
 WHEN latest.query_sql_text LIKE N'%INSERT INTO%dbo.AuditPosts%'
 AND latest.query_sql_text LIKE N'%Action = ''I''%'
 THEN N'INSERT INTO AuditPosts (Action=I)'
 WHEN latest.query_sql_text LIKE N'%INSERT INTO%dbo.AuditPosts%'
 AND latest.query_sql_text LIKE N'%Action = ''U''%'
 THEN N'INSERT INTO AuditPosts (Action=U)'
 WHEN latest.query_sql_text LIKE N'%INSERT INTO%dbo.AuditPosts%'
 AND latest.query_sql_text LIKE N'%Action = ''D''%'
 THEN N'INSERT INTO AuditPosts (Action=D)'
 ELSE LEFT(latest.query_sql_text, 80) + N'...'
 END,
 /* List only steps where execution count increased, in order, with execution counts */
 steps_used =
 STRING_AGG
 (
 step_list.step_description +
 N' (' +
 CONVERT(nvarchar(10), step_list.executions_added) +
 CASE
 WHEN step_list.executions_added = 1
 THEN N' exec'
 ELSE N' execs'
 END +
 N')',
 N', '
 )
 WITHIN GROUP
 (
 ORDER BY
 step_list.step_number ASC
 ),
 /* Sum execution counts across all plans for the latest step */
 total_executions =
 (
 SELECT
 SUM(qsh_final.count_executions)
 FROM dbo.QueryStoreHistory AS qsh_final
 WHERE qsh_final.query_id = latest.query_id
 AND qsh_final.context_settings_id = latest.context_settings_id
 AND qsh_final.step_number = latest.max_step_number
 ),
 total_compiles = latest.max_count_compiles
FROM
(
 /* Get latest state for each query_id and context_settings_id */
 SELECT
 qsh.query_id,
 qsh.context_settings_id,
 qsh.set_options_decoded,
 qsh.query_sql_text,
 max_step_number = MAX(qsh.step_number),
 max_count_compiles = MAX(qsh.count_compiles),
 row_number =
 ROW_NUMBER() OVER
 (
 PARTITION BY
 qsh.query_id,
 qsh.context_settings_id
 ORDER BY
 MAX(qsh.step_number) DESC
 )
 FROM dbo.QueryStoreHistory AS qsh
 GROUP BY
 qsh.query_id,
 qsh.context_settings_id,
 qsh.set_options_decoded,
 qsh.query_sql_text
) AS latest
/* Get only steps where execution count increased - aggregate by step_number first, summing across all plans */
CROSS APPLY
(
 SELECT
 step_agg.step_number,
 step_agg.step_description,
 step_agg.total_count_executions,
 prev_step.total_count_executions AS prev_count_executions,
 /* Calculate executions added in this step */
 executions_added =
 CASE
 WHEN prev_step.total_count_executions IS NULL
 THEN step_agg.total_count_executions
 ELSE step_agg.total_count_executions - prev_step.total_count_executions
 END
 FROM
 (
 /* Aggregate by step_number, summing execution counts across all plans */
 SELECT
 qsh2.step_number,
 qsh2.step_description,
 total_count_executions = SUM(qsh2.count_executions)
 FROM dbo.QueryStoreHistory AS qsh2
 WHERE qsh2.query_id = latest.query_id
 AND qsh2.context_settings_id = latest.context_settings_id
 GROUP BY
 qsh2.step_number,
 qsh2.step_description
 ) AS step_agg
 /* Get previous step's total execution count (summed across all plans) */
 /* Get the immediately previous step (highest step_number < current step) */
 OUTER APPLY
 (
 SELECT
 total_count_executions = SUM(qsh3.count_executions)
 FROM dbo.QueryStoreHistory AS qsh3
 WHERE qsh3.query_id = latest.query_id
 AND qsh3.context_settings_id = latest.context_settings_id
 AND qsh3.step_number =
 (
 SELECT
 MAX(qsh_prev.step_number)
 FROM dbo.QueryStoreHistory AS qsh_prev
 WHERE qsh_prev.query_id = latest.query_id
 AND qsh_prev.context_settings_id = latest.context_settings_id
 AND qsh_prev.step_number < step_agg.step_number
 )
 ) AS prev_step
 /* Only include steps where execution count increased (or first appearance) */
 WHERE (
 prev_step.total_count_executions IS NULL
 OR step_agg.total_count_executions > prev_step.total_count_executions
 )
) AS step_list
WHERE latest.row_number = 1
GROUP BY
 latest.query_id,
 latest.context_settings_id,
 latest.set_options_decoded,
 latest.query_sql_text,
 latest.max_step_number,
 latest.max_count_compiles
ORDER BY
 latest.query_id,
 latest.context_settings_id;
GO

SELECT
 qsq.query_id,
 qsq.context_settings_id,
 total_executions = SUM(qsrs.count_executions),
 total_compiles = MAX(qsq.count_compiles)
FROM sys.query_store_query AS qsq
JOIN sys.query_store_plan AS qsp
 ON qsp.query_id = qsq.query_id
JOIN sys.query_store_runtime_stats AS qsrs
 ON qsrs.plan_id = qsp.plan_id
WHERE qsq.object_id = OBJECT_ID(N'dbo.tr_Posts_Audit_No_Safeties', N'TR')
GROUP BY
 qsq.query_id,
 qsq.context_settings_id
ORDER BY
 qsq.query_id,
 qsq.context_settings_id;
GO

This summary shows:

Only steps where execution count increased for each query_id (steps_used), listed in order
Final execution and compile counts

Here’s the actual Query Store output from running the test procedure. Look for how query IDs are split by context settings and which steps use each query ID:

query id (context settings id)	implicit transaction?	query summary	steps used	total executions
3 (2)	OFF	UPDATE LastActivityDate	Step A: INSERT 10 rows (1 exec), Step L: DELETE 10 rows (1 exec)	2
4 (2)	OFF	INSERT INTO AuditPosts (Action=I)	Step A: INSERT 10 rows (1 exec), Step L: DELETE 10 rows (1 exec)	2
5 (2)	OFF	INSERT INTO AuditPosts (Action=U)	Step A: INSERT 10 rows (1 exec), Step L: DELETE 10 rows (1 exec)	2
6 (2)	OFF	INSERT INTO AuditPosts (Action=D)	Step A: INSERT 10 rows (1 exec), Step L: DELETE 10 rows (1 exec)	2
12 (7)	ON	UPDATE LastActivityDate	Step B: UPDATE Question 1 (Title) (1 exec), Step C: UPDATE Question 2 (CommentCount) (1 exec), Step D: UPDATE Question 3 (Body) (1 exec), Step E: UPDATE Question 4 (LastEditDate) (1 exec), Step F: UPDATE Question 5 (Score) (1 exec), Step G: UPDATE Answer 1 (OwnerUserId) (1 exec), Step H: UPDATE Answer 2 (CommentCount) (1 exec), Step I: UPDATE Answer 3 (Body) (1 exec), Step J: UPDATE Answer 4 (LastEditDate) (1 exec), Step K: UPDATE Answer 5 (Score) (1 exec), Step N: Dynamic SQL DELETE (1 exec)	11
13 (7)	ON	INSERT INTO AuditPosts (Action=I)	Step B: UPDATE Question 1 (Title) (1 exec), Step C: UPDATE Question 2 (CommentCount) (1 exec), Step D: UPDATE Question 3 (Body) (1 exec), Step E: UPDATE Question 4 (LastEditDate) (1 exec), Step F: UPDATE Question 5 (Score) (1 exec), Step G: UPDATE Answer 1 (OwnerUserId) (1 exec), Step H: UPDATE Answer 2 (CommentCount) (1 exec), Step I: UPDATE Answer 3 (Body) (1 exec), Step J: UPDATE Answer 4 (LastEditDate) (1 exec), Step K: UPDATE Answer 5 (Score) (1 exec), Step N: Dynamic SQL DELETE (1 exec)	11
14 (7)	ON	INSERT INTO AuditPosts (Action=U)	Step B: UPDATE Question 1 (Title) (1 exec), Step C: UPDATE Question 2 (CommentCount) (1 exec), Step D: UPDATE Question 3 (Body) (1 exec), Step E: UPDATE Question 4 (LastEditDate) (1 exec), Step F: UPDATE Question 5 (Score) (1 exec), Step G: UPDATE Answer 1 (OwnerUserId) (1 exec), Step H: UPDATE Answer 2 (CommentCount) (1 exec), Step I: UPDATE Answer 3 (Body) (1 exec), Step J: UPDATE Answer 4 (LastEditDate) (1 exec), Step K: UPDATE Answer 5 (Score) (1 exec), Step N: Dynamic SQL DELETE (1 exec)	11
15 (7)	ON	INSERT INTO AuditPosts (Action=D)	Step B: UPDATE Question 1 (Title) (1 exec), Step C: UPDATE Question 2 (CommentCount) (1 exec), Step D: UPDATE Question 3 (Body) (1 exec), Step E: UPDATE Question 4 (LastEditDate) (1 exec), Step F: UPDATE Question 5 (Score) (1 exec), Step G: UPDATE Answer 1 (OwnerUserId) (1 exec), Step H: UPDATE Answer 2 (CommentCount) (1 exec), Step I: UPDATE Answer 3 (Body) (1 exec), Step J: UPDATE Answer 4 (LastEditDate) (1 exec), Step K: UPDATE Answer 5 (Score) (1 exec), Step N: Dynamic SQL DELETE (1 exec)	11
27 (10)	ON	UPDATE LastActivityDate	Step M: MERGE statement (1 exec)	1
28 (10)	ON	INSERT INTO AuditPosts (Action=I)	Step M: MERGE statement (1 exec)	1
29 (10)	ON	INSERT INTO AuditPosts (Action=U)	Step M: MERGE statement (1 exec)	1
30 (10)	ON	INSERT INTO AuditPosts (Action=D)	Step M: MERGE statement (1 exec)	1
31 (11)	ON	UPDATE LastActivityDate	Step M: MERGE statement (1 exec)	1
32 (11)	ON	INSERT INTO AuditPosts (Action=I)	Step M: MERGE statement (1 exec)	1
33 (11)	ON	INSERT INTO AuditPosts (Action=U)	Step M: MERGE statement (1 exec)	1
34 (11)	ON	INSERT INTO AuditPosts (Action=D)	Step M: MERGE statement (1 exec)	1

Observations from the Query Store data:

Separate query_ids for different trigger operations: The same DML statement (e.g., INSERT 10 rows in Step A) appears across multiple query_ids (3, 4, 5, 6), each representing a different trigger operation type:
- Query ID 3: UPDATE LastActivityDate
- Query ID 4: INSERT INTO AuditPosts with Action=‘I’
- Query ID 5: INSERT INTO AuditPosts with Action=‘U’
- Query ID 6: INSERT INTO AuditPosts with Action=‘D’
This matches the trigger’s structure: it contains separate SQL statements for each operation type (one UPDATE for LastActivityDate, and separate INSERT statements for Action=‘I’, ‘U’, and ‘D’). Because the trigger uses no safety checks (no IF EXISTS blocks), all these statements are always present and execute based on their WHERE clause conditions (WHERE NOT EXISTS (SELECT 1/0 FROM deleted AS d) for INSERT operations, ISNULL comparisons for UPDATE operations, LEFT JOIN inserted ... WHERE i.Id IS NULL for DELETE operations), rather than being conditionally executed via IF statements.

Important note about Query Store execution counts: Query Store counts all statement executions in the trigger, even when WHERE clauses match 0 rows. For example, Query ID 3 (UPDATE LastActivityDate) shows execution during Step L DELETE, but this UPDATE affects 0 rows because it joins to the inserted table which is empty during DELETE operations. Query Store tracks the execution, but the statement does no actual work. This is why the test summary correctly notes that DELETE operations don’t update LastActivityDate, even though Query Store shows executions.
Multiple context settings created: Even though we didn’t vary user settings, Query Store created 4 different context_settings_id values:
- context_settings_id = 2 (IMPLICIT_TRANSACTIONS OFF): Used for Step A (INSERT 10 rows) and Step L (DELETE 10 rows)
- context_settings_id = 7 (IMPLICIT_TRANSACTIONS ON): Used for Steps B-K and N (individual UPDATEs and dynamic SQL DELETE)
- context_settings_id = 10 (IMPLICIT_TRANSACTIONS ON): Used for Step M (MERGE statement)
- context_settings_id = 11 (IMPLICIT_TRANSACTIONS ON): Also used for Step M (MERGE statement)
Interestingly, the MERGE statement (Step M) appears with two different context_settings_id values (10 and 11), both showing IMPLICIT_TRANSACTIONS ON. This happens even though we didn’t change SET options between operations. The MERGE fires the trigger twice (once for the UPDATE action, once for the INSERT action), and Query Store assigns different context_settings_id values to each trigger execution from that single MERGE statement.

The same DML statement creates multiple query_ids (one per trigger operation) and multiple context_settings_id values, even when you don’t change any settings. Data ends up spread across all these query_ids and context settings, so it’s easy to miss executions if your queries don’t aggregate across everything.

XEvents Event File Session on sqlserver.module_start

Let’s create an Extended Events session to capture when the trigger module starts. This approach is based on Tomáš Zíka’s answer from straightforwardsql.com, using sqlserver.module_start.

This is created with MAX_DISPATCH_LATENCY = 5 SECONDS for convenience in testing, for production code you generally want to use a longer amount.

/* Drop if exists and create idempotent session */
IF EXISTS
(
 SELECT
 1/0
 FROM sys.server_event_sessions AS ses
 WHERE ses.name = N'trigger_module_start'
)
BEGIN
 ALTER EVENT SESSION [trigger_module_start]
 ON SERVER
 STATE = STOP;

 DROP EVENT SESSION [trigger_module_start]
 ON SERVER;
END;
GO

CREATE EVENT SESSION [trigger_module_start]
ON SERVER
 ADD EVENT sqlserver.module_start
 (
 ACTION
 (
 sqlserver.sql_text,
 sqlserver.tsql_stack,
 sqlserver.client_app_name,
 sqlserver.database_name,
 sqlserver.session_id,
 sqlserver.username
 )
 WHERE
 (
 [sqlserver].[database_name] = N'StackOverflow2013'
 AND [object_name] = N'tr_Posts_Audit_No_Safeties'
 )
 )
 ADD TARGET package0.event_file
 (
 SET filename = N'C:\Temp\trigger_module_start.xel',
 max_file_size = 50,
 max_rollover_files = 5
 )
WITH
(
 MAX_MEMORY = 4096 KB,
 MAX_DISPATCH_LATENCY = 5 SECONDS,
 EVENT_RETENTION_MODE = ALLOW_MULTIPLE_EVENT_LOSS
);
GO

ALTER EVENT SESSION [trigger_module_start]
ON SERVER
STATE = START;
GO

XEvents Event File Session on sqlserver.module_end

Let’s also create an Extended Events session to capture when the trigger module ends. Unless you are concerned about lookign for trigger failures, this is arguably more useful than module_start, as it allows us to capture duration and row_count metrics.

This is created with MAX_DISPATCH_LATENCY = 5 SECONDS for convenience in testing, for production code you generally want to use a longer amount.

/* Drop if exists and create idempotent session */
IF EXISTS
(
 SELECT
 1/0
 FROM sys.server_event_sessions AS ses
 WHERE ses.name = N'trigger_module_end'
)
BEGIN
 ALTER EVENT SESSION [trigger_module_end]
 ON SERVER
 STATE = STOP;

 DROP EVENT SESSION [trigger_module_end]
 ON SERVER;
END;
GO

CREATE EVENT SESSION [trigger_module_end]
ON SERVER
 ADD EVENT sqlserver.module_end
 (
 ACTION
 (
 sqlserver.sql_text,
 sqlserver.tsql_stack,
 sqlserver.client_app_name,
 sqlserver.database_name,
 sqlserver.session_id,
 sqlserver.username
 )
 WHERE
 (
 [sqlserver].[database_name] = N'StackOverflow2013'
 AND [object_name] = N'tr_Posts_Audit_No_Safeties'
 )
 )
 ADD TARGET package0.event_file
 (
 SET filename = N'C:\Temp\trigger_module_end.xel',
 max_file_size = 50,
 max_rollover_files = 5
 )
WITH
(
 MAX_MEMORY = 4096 KB,
 MAX_DISPATCH_LATENCY = 5 SECONDS,
 EVENT_RETENTION_MODE = ALLOW_MULTIPLE_EVENT_LOSS
);
GO

ALTER EVENT SESSION [trigger_module_end]
ON SERVER
STATE = START;
GO

XEvents Event File Session on Stored Procedure Statement Completions

Now let’s create a session to capture statement completions within the trigger. We’ll capture sp_statement_completed filtered to only the trigger object to see which statements execute inside the trigger.

Note: To filter sp_statement_completed events by object name, you must include SET collect_object_name = (1) in the event definition. When efficiency is important, filter on object_id instead of object_name (object_id is always available and doesn’t require the collect_object_name setting).

This is created with MAX_DISPATCH_LATENCY = 5 SECONDS for convenience in testing, for production code you generally want to use a longer amount.

IF EXISTS
(
 SELECT
 1/0
 FROM sys.server_event_sessions AS ses
 WHERE ses.name = N'trigger_completions'
)
BEGIN
 ALTER EVENT SESSION [trigger_completions]
 ON SERVER
 STATE = STOP;

 DROP EVENT SESSION [trigger_completions]
 ON SERVER;
END;
GO

CREATE EVENT SESSION [trigger_completions]
ON SERVER
 ADD EVENT sqlserver.sp_statement_completed
 (
 SET collect_object_name = (1)
 ACTION
 (
 sqlserver.client_app_name,
 sqlserver.database_name,
 sqlserver.username,
 sqlserver.session_id,
 sqlserver.sql_text
 )
 WHERE
 (
 [sqlserver].[database_name] = N'StackOverflow2013'
 AND [object_name] = N'tr_Posts_Audit_No_Safeties'
 )
 )
 ADD TARGET package0.event_file
 (
 SET filename = N'C:\Temp\trigger_completions.xel',
 max_file_size = 50,
 max_rollover_files = 5
 )
WITH
(
 MAX_MEMORY = 4096 KB,
 MAX_DISPATCH_LATENCY = 5 SECONDS,
 EVENT_RETENTION_MODE = ALLOW_MULTIPLE_EVENT_LOSS
);
GO

ALTER EVENT SESSION [trigger_completions]
ON SERVER
STATE = START;
GO

XEvents Histogram Session on sp_statement_completed

Now let’s create a histogram session that tracks statement executions within the trigger. This histogram counts executions per unique statement, giving us an aggregated view of which statements execute most frequently.

This uses a synchronous target (histogram) rather than an asynchronous target (event_file), so it aggregates data in memory rather than writing individual events to disk.

IF EXISTS
(
 SELECT
 1/0
 FROM sys.server_event_sessions AS ses
 WHERE ses.name = N'trigger_statement_histogram'
)
BEGIN
 ALTER EVENT SESSION [trigger_statement_histogram]
 ON SERVER
 STATE = STOP;

 DROP EVENT SESSION [trigger_statement_histogram]
 ON SERVER;
END;
GO

CREATE EVENT SESSION [trigger_statement_histogram]
ON SERVER
 ADD EVENT sqlserver.sp_statement_completed
 (
 SET collect_object_name = (1)
 WHERE
 (
 [sqlserver].[database_name] = N'StackOverflow2013'
 AND [object_name] = N'tr_Posts_Audit_No_Safeties'
 )
 )
 ADD TARGET package0.histogram
 (
 SET filtering_event_name = N'sqlserver.sp_statement_completed',
 slots = 256,
 source = N'object_name',
 source_type = 0
 )
WITH
(
 MAX_MEMORY = 4096 KB,
 EVENT_RETENTION_MODE = ALLOW_MULTIPLE_EVENT_LOSS
);
GO

ALTER EVENT SESSION [trigger_statement_histogram]
ON SERVER
STATE = START;
GO

Run our Test Procedure Again, Then Query All Trace Files

Now let’s run our test procedure again and query the Extended Events data.

Because the Extended Events session is configured with MAX_DISPATCH_LATENCY = 5 SECONDS, we wait 5 seconds after running the test procedure before querying the trace files.

Run Test Procedure

/* Run test - audit table will accumulate rows from all runs */
/* Ensure consistent SET options for Query Store tracking */
SET ANSI_NULLS ON;
SET QUOTED_IDENTIFIER ON;
SET ARITHABORT ON;
SET NUMERIC_ROUNDABORT OFF;
SET IMPLICIT_TRANSACTIONS OFF;
GO

EXECUTE dbo.sp_TestTriggerFires;
GO

Because the Extended Events session is configured with MAX_DISPATCH_LATENCY = 5 SECONDS, we need to wait at least 5 seconds after running the test procedure before querying the trace files. This ensures events have been flushed from the buffer to disk.

Query module_start Events

SELECT
 event_data.value('(event/@name)[1]', 'nvarchar(50)') AS event_name,
 event_data.value('(event/@timestamp)[1]', 'datetime2') AS event_timestamp,
 event_data.value('(event/data[@name="object_name"]/value)[1]', 'nvarchar(128)') AS object_name,
 event_data.value('(event/data[@name="object_id"]/value)[1]', 'integer') AS object_id,
 calling_code = event_data.value('(event/action[@name="sql_text"]/value)[1]', 'nvarchar(max)'),
 tsql_stack = CONVERT(nvarchar(max), event_data.query('(event/action[@name="tsql_stack"]/value)')),
 client_app = event_data.value('(event/action[@name="client_app_name"]/value)[1]', 'nvarchar(128)'),
 session_id = event_data.value('(event/action[@name="session_id"]/value)[1]', 'integer'),
 username = event_data.value('(event/action[@name="username"]/value)[1]', 'nvarchar(128)'),
 full_event_xml = event_data
FROM
(
 SELECT
 CAST(event_data AS xml) AS event_data
 FROM sys.fn_xe_file_target_read_file
 (
 N'C:\Temp\trigger_module_start*.xel',
 NULL,
 NULL,
 NULL
 )
) AS x
ORDER BY
 event_data.value('(event/@timestamp)[1]', 'datetime2');
GO

🔥 Note: The query includes the tsql_stack action which contains the call stack with sql_handle values that can be used to identify the DML statement that fired the trigger. To parse the tsql_stack and retrieve statement text from the plan cache, see Tomáš Zíka's answer on Stack Overflow for detailed instructions.

Query module_end Events

Module_end events have duration and row_count columns, so those are included here. Duration is in microseconds on SQL Server 2025.

SELECT
 event_data.value('(event/@name)[1]', 'nvarchar(50)') AS event_name,
 event_data.value('(event/@timestamp)[1]', 'datetime2') AS event_timestamp,
 event_data.value('(event/data[@name="object_name"]/value)[1]', 'nvarchar(128)') AS object_name,
 event_data.value('(event/data[@name="object_id"]/value)[1]', 'integer') AS object_id,
 event_data.value('(event/data[@name="duration"]/value)[1]', 'integer') AS duration,
 event_data.value('(event/data[@name="row_count"]/value)[1]', 'integer') AS row_count,
 calling_code = event_data.value('(event/action[@name="sql_text"]/value)[1]', 'nvarchar(max)'),
 tsql_stack = CONVERT(nvarchar(max), event_data.query('(event/action[@name="tsql_stack"]/value)')),
 client_app = event_data.value('(event/action[@name="client_app_name"]/value)[1]', 'nvarchar(128)'),
 session_id = event_data.value('(event/action[@name="session_id"]/value)[1]', 'integer'),
 username = event_data.value('(event/action[@name="username"]/value)[1]', 'nvarchar(128)'),
 full_event_xml = event_data
FROM
(
 SELECT
 CAST(event_data AS xml) AS event_data
 FROM sys.fn_xe_file_target_read_file
 (
 N'C:\Temp\trigger_module_end*.xel',
 NULL,
 NULL,
 NULL
 )
) AS x
ORDER BY
 event_data.value('(event/@timestamp)[1]', 'datetime2');
GO

Query Completion Events - sp_statement_Completed for Trigger Statements

SELECT
 event_data.value('(event/@name)[1]', 'nvarchar(50)') AS event_name,
 event_data.value('(event/@timestamp)[1]', 'datetime2') AS event_timestamp,
 event_data.value('(event/data[@name="duration"]/value)[1]', 'bigint') AS duration_microseconds,
 event_data.value('(event/data[@name="cpu_time"]/value)[1]', 'bigint') AS cpu_time_microseconds,
 event_data.value('(event/data[@name="logical_reads"]/value)[1]', 'bigint') AS logical_reads,
 event_data.value('(event/data[@name="statement"]/value)[1]', 'nvarchar(max)') AS statement_text,
 event_data.value('(event/action[@name="sql_text"]/value)[1]', 'nvarchar(max)') AS sql_text,
 event_data.value('(event/action[@name="database_name"]/value)[1]', 'nvarchar(128)') AS database_name,
 object_name = event_data.value('(event/data[@name="object_name"]/value)[1]', 'nvarchar(128)'),
 object_id = event_data.value('(event/data[@name="object_id"]/value)[1]', 'integer'),
 client_app = event_data.value('(event/action[@name="client_app_name"]/value)[1]', 'nvarchar(128)'),
 session_id = event_data.value('(event/action[@name="session_id"]/value)[1]', 'integer'),
 username = event_data.value('(event/action[@name="username"]/value)[1]', 'nvarchar(128)'),
 full_event_xml = event_data
FROM
(
 SELECT
 CAST(event_data AS xml) AS event_data
 FROM sys.fn_xe_file_target_read_file
 (
 N'C:\Temp\trigger_completions*.xel',
 NULL,
 NULL,
 NULL
 )
) AS x
ORDER BY
 event_timestamp;
GO

Query Aggregate sp_statement_completed Events

To get aggregated metrics like total CPU time and total logical reads per statement, query the event_file target from trigger_completions and aggregate.

SELECT
 statement_text,
 execution_count = COUNT_BIG(*),
 total_cpu_time_ms = SUM(cpu_time_microseconds) / 1000.0,
 avg_cpu_time_ms = AVG(cpu_time_microseconds) / 1000.0,
 total_logical_reads = SUM(logical_reads),
 avg_logical_reads = AVG(logical_reads),
 total_duration_ms = SUM(duration_microseconds) / 1000.0,
 avg_duration_ms = AVG(duration_microseconds) / 1000.0
FROM
(
 SELECT
 statement_text = event_data.value('(event/data[@name="statement"]/value)[1]', 'nvarchar(max)'),
 cpu_time_microseconds = event_data.value('(event/data[@name="cpu_time"]/value)[1]', 'bigint'),
 logical_reads = event_data.value('(event/data[@name="logical_reads"]/value)[1]', 'bigint'),
 duration_microseconds = event_data.value('(event/data[@name="duration"]/value)[1]', 'bigint')
 FROM
 (
 SELECT
 CAST(event_data AS xml) AS event_data
 FROM sys.fn_xe_file_target_read_file
 (
 N'C:\Temp\trigger_completions*.xel',
 NULL,
 NULL,
 NULL
 )
 ) AS x
) AS statement_data
GROUP BY
 statement_text
ORDER BY
 total_cpu_time_ms DESC;
GO

Query histogram Target for Statement Execution Counts

SELECT
 slot_data.value('(value)[1]', 'nvarchar(max)') AS statement_text,
 slot_data.value('(@count)[1]', 'bigint') AS execution_count
FROM
(
 SELECT
 CAST(target_data AS xml) AS target_data
 FROM sys.dm_xe_session_targets AS xst
 JOIN sys.dm_xe_sessions AS xs
 ON xs.address = xst.event_session_address
 WHERE xs.name = N'trigger_statement_histogram'
 AND xst.target_name = N'histogram'
) AS histogram_data
CROSS APPLY target_data.nodes('//HistogramTarget/Slot') AS histogram_slots(slot_data)
ORDER BY
 slot_data.value('(@count)[1]', 'bigint') DESC;
GO

Extended Events Observations

Comparing Extended Events data with the audit table and Query Store:

Trigger execution counts:

Extended Events module_start and module_end captured 15 trigger executions, even though the test procedure executed only 14 DML statements. The discrepancy comes from MERGE: as Paul White explains, MERGE can fire triggers multiple times—once for each action type (INSERT, UPDATE, DELETE) that executes. Our MERGE statement (Step M) performs both an INSERT (WHEN NOT MATCHED BY TARGET) and an UPDATE (WHEN MATCHED), so the trigger fires twice for that single MERGE statement.

Query Store execution counts align with this when aggregated across all query_ids and context_settings_id values:

Query IDs 3-6 (context_settings_id 2): 2 executions total (Step A INSERT + Step L DELETE)
Query IDs 12-15 (context_settings_id 7): 11 executions total (Steps B-K individual UPDATEs + Step N dynamic SQL DELETE)
Query IDs 27-34 (context_settings_id 10, 11): 2 executions total (Step M MERGE fires trigger twice, creating two different context_settings_id values)
Total: 2 + 11 + 2 = 15 trigger executions

Statement-level vs query-level granularity:

Extended Events sp_statement_completed captured 75 individual statement executions within the trigger. This is statement-level detail—each SQL statement that executed inside the trigger, not the trigger executions themselves. Each trigger execution contains multiple statements (SET NOCOUNT ON, UPDATE LastActivityDate, INSERT INTO AuditPosts statements).

Query Store tracks query-level executions aggregated by query_id and context_settings_id. The same trigger operation (like “UPDATE LastActivityDate”) appears across multiple query_ids depending on context settings, requiring careful aggregation to get accurate counts.

The histogram target shows execution counts per unique statement, matching the event_file counts when grouped by statement text.

Row-level results:

The audit table shows 104 rows inserted—this is row-level detail, a different granularity entirely from both Extended Events statement counts and Query Store query counts.

Cleanup

This script tidies everything up:

/* Stop and drop Extended Events sessions */
IF EXISTS
(
 SELECT
 1/0
 FROM sys.server_event_sessions AS ses
 WHERE ses.name = N'trigger_module_start'
)
BEGIN
 ALTER EVENT SESSION [trigger_module_start]
 ON SERVER
 STATE = STOP;

 DROP EVENT SESSION [trigger_module_start]
 ON SERVER;
END;
GO

IF EXISTS
(
 SELECT
 1/0
 FROM sys.server_event_sessions AS ses
 WHERE ses.name = N'trigger_module_end'
)
BEGIN
 ALTER EVENT SESSION [trigger_module_end]
 ON SERVER
 STATE = STOP;

 DROP EVENT SESSION [trigger_module_end]
 ON SERVER;
END;
GO

IF EXISTS
(
 SELECT
 1/0
 FROM sys.server_event_sessions AS ses
 WHERE ses.name = N'trigger_completions'
)
BEGIN
 ALTER EVENT SESSION [trigger_completions]
 ON SERVER
 STATE = STOP;

 DROP EVENT SESSION [trigger_completions]
 ON SERVER;
END;
GO

IF EXISTS
(
 SELECT
 1/0
 FROM sys.server_event_sessions AS ses
 WHERE ses.name = N'trigger_statement_histogram'
)
BEGIN
 ALTER EVENT SESSION [trigger_statement_histogram]
 ON SERVER
 STATE = STOP;

 DROP EVENT SESSION [trigger_statement_histogram]
 ON SERVER;
END;
GO

/* Clean up test objects */
IF OBJECT_ID(N'dbo.tr_Posts_Audit_No_Safeties', N'TR') IS NOT NULL
BEGIN
 DROP TRIGGER dbo.tr_Posts_Audit_No_Safeties;
END;
GO

IF OBJECT_ID(N'dbo.sp_TestTriggerFires', N'P') IS NOT NULL
BEGIN
 DROP PROCEDURE dbo.sp_TestTriggerFires;
END;
GO

IF OBJECT_ID(N'dbo.sp_CaptureQueryStoreHistory', N'P') IS NOT NULL
BEGIN
 DROP PROCEDURE dbo.sp_CaptureQueryStoreHistory;
END;
GO

IF OBJECT_ID(N'dbo.QueryStoreHistory', N'U') IS NOT NULL
BEGIN
 DROP TABLE dbo.QueryStoreHistory;
END;
GO

IF OBJECT_ID(N'dbo.AuditPosts', N'U') IS NOT NULL
BEGIN
 DROP TABLE dbo.AuditPosts;
END;
GO

What We Observed About Tracing Trigger Executions

Both Query Store and Extended Events can trace trigger executions, but they work at different levels of detail and have different strengths.

Query Store tracks query-level executions, which means trigger code gets split across multiple query_ids (one per SQL statement in the trigger). The same DML statement can create multiple query_ids and context_settings_id values, even when you don’t change any session settings. You need to aggregate across all query_ids and context settings to get accurate execution counts. The upside is that Query Store automatically captures execution plans, runtime statistics, and performance metrics without any additional setup.

Extended Events gives you more control over what you capture. The sqlserver.module_start event captures each time the trigger module starts executing, giving you a clean count of trigger executions. The sp_statement_completed event captures individual statement executions within the trigger, showing you exactly which statements ran and their performance metrics. Extended Events requires more setup and configuration, but it provides statement-level detail that Query Store doesn’t expose directly.

To filter sp_statement_completed events by object name, you must include SET collect_object_name = (1) in the event definition. For better performance, you can filter on object_id instead of object_name (object_id is always available and doesn’t require the collect_object_name setting).

Key differences we found:

Execution counts: Extended Events module_start showed 15 trigger executions, which matched Query Store’s aggregated counts once we accounted for MERGE firing the trigger twice.
Granularity: Extended Events sp_statement_completed captured 75 individual statement executions within the trigger, while Query Store tracked query-level executions aggregated by query_id. The audit table showed 104 rows inserted, which is yet another level of detail (row-level results).
Data organization: Query Store data spreads across multiple query_ids and context_settings_id values, making it easy to miss executions if your queries don’t aggregate properly. Extended Events gives you event-level data that’s easier to filter and analyze, but requires more manual setup.

For tracing trigger executions, Extended Events provides cleaner, more direct counts and statement-level detail. Query Store works well for performance analysis of the queries within triggers, but requires careful aggregation to get accurate execution counts.

Static Site Improvements: Scheduled Posts, Preview Builds, Robots Writing the CSS

help@KendraLittle.com (Kendra Little) — Mon, 22 Dec 2025 00:00:00 +0000

I’ve standardized design, upgraded components, added client-side search, and configured browser caching for this site over the last few weeks. I’ve also adopted a workflow that helps me capture ideas, test changes, and get posts published more easily.

I’m not a web developer, I’m a database nerd. I’m don’t want to be great at CSS. I got this all done with a personal Cursor subscription and an obsessive urge to tag and tidy all my old posts. AI tooling these days makes static websites easier than ever to set up, design, and improve.

If you’re open to AI tooling and want to create a website (or upgrade one you already have), a static site is a great bet.

🔥 Related posts: I've written about my static site journey before. Check out Moving From WordPress to an Azure Static Site with Hugo for my migration story, and Adding a Netlify Contact Form to a Hugo Static Site for details on moving to Netlify.

Recap of My Site Setup

KendraLittle.com is a static site built with Hugo, a static site generator. I write posts in Markdown, Hugo builds them into HTML files, and Netlify hosts and deploys the site. There’s no database, no server-side code, and no Content Management System, just files in a free GitHub repository and Netlify.

I currently pay for a Netlify subscription and a Cursor subscription. It’s a hobby for me, I want some fun things, so those costs are worth it to me. (Netlify has a good free tier, but occasionally I’m lucky enough to exceed its bandwidth costs. And I feel like I get good value for what I pay for.)

This setup works well for a technical blog. It’s fast, secure, and easy to maintain.

And I have never missed WordPress.

Component Upgrades

I upgraded the major frontend components to bring the site up to date:

Bootstrap: Upgraded from 3.3.7 to 5.3.3
Hugo: Upgraded from 0.124.1 to 0.152.2

The Hugo upgrade here was a big deal. There were some important fixes between those two versions, and I’d struggled to do this upgrade on my own before. I knew it was important to complete this before making more changes.

These upgrades required template changes. Bootstrap 5 changed class names (navbar-toggle → navbar-toggler, data-toggle → data-bs-toggle). Hugo 0.152.2 deprecated some template functions, so I updated those too. Cursor helped identify all the places that needed changes, which made the upgrade process manageable.

It wasn’t bad this time at all, especially with the help of…

Preview Deployments

Every time I open a pull request or push a change to a branch with a pull request, Netlify creates a preview deployment. I can see how the site looks and behaves before merging to main. This is especially valuable for CSS changes: I can test button heights, spacing, and mobile layouts in a real deployment without affecting the live site.

It’s basically a staging environment that’s automatically created for every pull request, and you can access it on different devices. (This post is not sponsored by Netlify, but this database person sure loves a staging site. They include this in the free plan, by the way.)

Local Development Workflow

I’m currently using Windows for my personal nerdery, so I have a PowerShell script that runs Hugo with specific flags for local development:

--disableFastRender to ensure CSS changes are visible immediately
--noHTTPCache to prevent browser caching issues
--buildFuture to preview future-dated posts

The script also syncs CSS files before starting the server and optionally builds the search index. These customizations make local development smoother: I can see CSS changes right away– at least usually. Admittedly, there are some times when I still have to do a full build of the site to see changes and I don’t fully understand why, but it’s not yet been a big enough problem to dive into.

Here’s the Hugo command the script runs:

hugo server --port 1313 --bind 127.0.0.1 --poll 700ms --disableFastRender --noHTTPCache --buildFuture

The --poll flag helps with reliable file watching on Windows, and --buildFuture lets me preview posts scheduled for future dates.

Standardized Buttons and Forms

Before, buttons and form fields had wildly inconsistent heights, padding, and styling. I had a hard time fixing it.

I used Cursor to find all button definitions and help standardize them. This cut down on the repetitive work, and chatting with the robot helped me figure out how to troubleshoot visual issues in Firefox somewhat efficiently (well, sometimes).

I standardized everything to a 38px height:

Buttons: All buttons (navigation, forms, sidebars) are 38px tall with consistent padding
Form fields: All input fields, selects, and textareas match the 38px height
Navigation links: Navbar links use the same 8px horizontal padding as buttons for visual consistency

It ain’t perfect, but it’s a lot better.

Mobile Improvements

I added mobile optimizations for screens under 768px:

Touch targets: Buttons and form fields meet the 44px minimum for easy tapping
Font sizes: Set to 16px minimum to prevent iOS auto-zoom on input focus
Navigation: Fixed navbar visibility and mobile menu behavior
Layout: Reduced padding and improved spacing for small screens
Typography: Scaled headings appropriately for mobile readability

These changes make the site easier to use on phones and tablets.

Client-Side Search with Pagefind

I added Pagefind for client-side search. It indexes the built HTML files and provides search entirely in the browser, no backend needed. Here’s my new search page, and there’s a search form in the footer, too.

How it works:

Pagefind indexes the site after Hugo builds
Search happens in the browser using the index files
Results are sorted by date (most recent first)

Why this works for static sites:

No server-side search needed
Fast search results (everything happens client-side)
Works with static site hosting (no backend required)

GitHub Issues for Blog Post Ideas and Bugs

I’m now using GitHub Issues to track bugs and planned blog posts.

The most important thing about this is that I can quickly create a GitHub issue on my phone for a blog post idea, wherever I am when I think about it. Or I can screenshot a bug on the site and create an issue for that, then forget it for the moment.

Years ago, when I was a part of Brent Ozar Unlimited, I had a big old list of blog post ideas in a shared GitHub repo with the team that I could pull from anytime I was in the mood to write. I loved it. It feels good to be getting back to that, even if it’s just me.

Automatic Issue Syncing

I created git hooks that automatically sync GitHub issues to local markdown files:

post-checkout hook: Syncs issues when you switch branches
post-merge hook: Syncs issues after pulling or merging

The hooks run a Python script that fetches issue data from the GitHub API and saves it to issues/issue-{number}.md. This lets me reference issues locally without opening GitHub in a browser.

I’m sure there are extensions and various other ways to do this, but I like markdown files, OK? My cursorrules file knows where my issues are, so I can also use prompts like, “new branch to write post in issue 29 to publish on Jan 10,” and it can get my workflow started and have an idea what basic things to put in the front matter of the new file it creates for me.

I may switch this to a different mechanism than a git hook. Git is complex enough. But it’s good enough for now.

🔥 Front matter is metadata at the top of a Markdown file, enclosed in --- markers. It contains information like the post title, date, categories, and tags. Hugo uses this metadata to generate the site structure and HTML.

Standardizing Categories and Tags

I spent a good chunk of time cleaning up categories and tags across all my old blog posts. I had inconsistent taxonomy: tons of posts used “sql” as a category, many posts were missing tags entirely.

What I standardized:

Categories: Established 9 main categories (like query-performance, indexing, database-administration) and moved old posts to the right categories. I also added azure-sql as a new category for cloud database content.
Tags: Cleaned up generic tags and replaced them with specific technical terms. Each post now has 3-10 meaningful tags that reflect what’s actually in the content. You can now find all my love letters to azure-sql-managed-instance all in one place.
Format: Standardized everything to lowercase with hyphens (e.g., query-store not Query Store or query_store).

This was not something that Cursor made super easy. I have 580+ posts at this point, and I didn’t feel like building a whole sentiment/content analysis machine. A few attempts to use python scripts to generate the tags didn’t go well, the tags were often ridiculous. Cursor doesn’t love going through that many posts manually, either.

I worked through a fair amount of this while cooking things for Thanksgiving and doing other things. Not letting perfect be the enemy of good helped get me through it, and I’m pretty happy with where we landed.

Future-Dated Posts for Scheduling

Hugo supports future-dated posts. I can set a post’s date in the future, and Hugo won’t publish it until that date arrives.

How it works:

Set the date in a post’s front matter to a future date.
Hugo won’t include the post in builds until after the build runs on a computer where it’s that date or later.
Netlify automatically rebuilds KendraLittle.com daily (more on that next), so future posts publish automatically.

Here’s an example of front matter for a scheduled post:

---
title: "My Future Post"
date: 2026-01-10
draft: false
---

This is a simple way to scheduling posts in advance. The static site generator (Hugo) handles the scheduling logic, and automated builds handle the publishing.

Netlify Automated Builds

Netlify automatically builds and deploys the site when I push to the main branch. It also runs scheduled builds twice daily.

How scheduled builds work:

Netlify’s scheduled function (configured under netlify/functions/) triggers at set times using a cron schedule
The function calls a Netlify build hook URL, which triggers a new build
Hugo builds the static site (including all posts with the date set to the current date or prior)
Pagefind indexes the site for search
Netlify deploys the result

Setting up the build trigger: I created a Netlify scheduled function that uses the @netlify/functions package’s schedule helper. The function is configured with a cron expression in netlify.toml and calls a build hook URL that Netlify generates. When you POST to that URL, it triggers a new deployment.

Here’s the scheduled function code:

import fetch from 'node-fetch'
import { schedule } from '@netlify/functions'

const BUILD_HOOK = 'https://api.netlify.com/build_hooks/YOUR_HOOK_ID'

// Runs twice daily: 11 AM UTC (6 AM EST) and 1 AM UTC (8 PM EST)
const handler = schedule('0 11,1 * * *', async () => {
 const response = await fetch(BUILD_HOOK, {
 method: 'POST'
 })

 return {
 statusCode: 200,
 body: JSON.stringify({ message: 'Deploy triggered successfully' })
 }
})

export { handler }

The cron expression '0 11,1 * * *' runs at 11 AM UTC and 1 AM UTC daily. You configure this in netlify.toml:

[functions.scheduled-deploy]
 schedule = "0 11,1 * * *"

Cache Headers

I’ve configured cache headers in netlify.toml to improve performance. Static assets (CSS, JavaScript, images, fonts) are cached for 1 year with the immutable flag: these files have versioned filenames from Hugo’s asset pipeline, so aggressive caching is safe. HTML files are cached for 1 hour with must-revalidate, which balances performance with the need to show updated content.

Here’s the cache header configuration in netlify.toml:

# Static assets - cache for 1 year (with versioning in filename)
[[headers]]
 for = "/*.css"
 [headers.values]
 Cache-Control = "public, max-age=31536000, immutable"

[[headers]]
 for = "/*.js"
 [headers.values]
 Cache-Control = "public, max-age=31536000, immutable"

[[headers]]
 for = "/images/*"
 [headers.values]
 Cache-Control = "public, max-age=31536000, immutable"

# HTML files - cache for 1 hour with revalidation
[[headers]]
 for = "/*.html"
 [headers.values]
 Cache-Control = "public, max-age=3600, must-revalidate"

The site also uses a service worker for offline support and client-side caching. Initially, it was using “Cache First” for images even during local development, which meant updated images wouldn’t show up in preview until I cleared the browser cache. I updated the service worker to detect when it’s running on localhost and use “Network First” for images in development, so image updates appear immediately. In production, it still uses “Cache First” for better performance.

Want to Build Your Own?

Here’s what I’m currently using and how to get started:

Core components:

Hugo (static site generator) - version 0.152.2
Universal theme (Hugo theme - includes Bootstrap 5.3.3 via CDN)
Pagefind (client-side search - add after you have content)
Hosting: Netlify, GitHub Pages, or Cloudflare Pages (all have free tiers)

Getting started:

Create a new GitHub repository for your site
Install Hugo: brew install hugo (Mac) or choco install hugo (Windows) or download from hugo.io
Clone your repo and create a new site: git clone <your-repo-url> && cd <repo-name> && hugo new site .
Create a .gitignore file with public/ and .hugo_build.lock (Hugo generates these, don’t commit them)
Choose a theme: Browse Hugo themes. To use the Universal theme, I add it as a git submodule: git submodule add https://github.com/devcows/hugo-universal-theme themes/universal
Configure config.toml with theme = "universal" (or your chosen theme name)
Create your first post: hugo new blog/my-first-post.md
Run locally: hugo server (view at http://localhost:1313)
Commit and push: git add . && git commit -m "Initial Hugo site" && git push
Deploy: Choose a hosting option (all have free tiers):

Netlify: Connect the repo in Netlify (includes preview deployments for pull requests)
GitHub Pages: Enable Pages in your repo settings and configure GitHub Actions to build and deploy
Cloudflare Pages: Free hosting with global CDN and automatic deployments

The Hugo documentation is excellent, and the Universal theme has good examples to get you started. Add Pagefind for search after you have content, and customize from there.

What About Jekyll?

Jekyll is another popular static site generator, and it’s the default for GitHub Pages. The main difference is that Jekyll is Ruby-based while Hugo is Go-based. Jekyll has a larger plugin ecosystem, while Hugo uses Go templates. Jekyll has big fans and a huge community. I’ve experimented with it recently on a new site and I really like it.

For now, I’m resisting the urge to give this site a Jekyll makeover, but if it still seems like a fun idea in a couple of months I may give it a shot.

How to Filter RPC_Completed Events in Extended Events: SQL Server Guide

help@KendraLittle.com (Kendra Little) — Mon, 15 Dec 2025 00:00:00 +0000

The rpc_completed event in Extended Events is useful when troubleshooting SQL Server performance. It captures detailed information about Remote Procedure Calls: that means stored procedure executions, including the calls to sp_executesql often used by applications (including Entity Framework) to run parameterized queries against SQL Server. The output for rpc_completed includes the parameters that were specified along with values provided, and the CPU time, logical reads, and duration used by the query.

It can be frustrating to figure out how to filter this in Extended Events. Struggling with this is one of the primary reasons I sometimes use ye Olde Profiler for initial investigations and to speedily observe something in SQL Server.

Here is my survival guide to filtering rpc_completed, which makes using XEvents suck less.

Surely ChatGPT or Claude Make Scripting this Easier?

As of this writing, they aren’t great at scripting Extended Event sessions with working filters. As I’ll dig into below, a lot of details of Extended Events aren’t documented clearly, the user is left to query them from a SQL Server instance. LLMs do what they can from published walkthroughs and StackOverflow answers, but they end up trying to use fields and filters for rpc_completed from other events which don’t work. They also tend to hallucinate things like whether you can filter on a simple field for schema name on rpc_completed. (You can’t.)

Possibly this post will make them a bit better by giving them more examples to scrape and work with. Good luck, robots.

sp_HumanEvents Can Save Time

If you want a simpler way to create Extended Events sessions with sensible defaults and filtering, check out Erik Darling’s sp_HumanEvents, a stored procedure that creates Extended Events sessions for common troubleshooting scenarios.

Why RPC_Completed Events Are Useful for Troubleshooting

The rpc_completed event has a key advantage over sql_batch_completed: it always captures parameter values for stored procedures and parameterized queries.

This makes it more useful when you want to set up code to reproduce a problem. It also makes it useful for troubleshooting parameter sniffing problems, where you need to see which specific parameter values are being passed in to run queries, as this isn’t captured in Query Store.

Data Fields in the RPC_Completed Event (and How to Query Them)

The rpc_completed event captures several useful data fields automatically as part of the event payload.

🔥 Beware: There are docs pages for Profiler events, like this one for RPC:Completed Event Class, but the column list and their names for Profiler Event Classes are different than what you get in Extended Events. Using this documentation to write XEvents scripts is a gotcha that will waste time.

You can query the available fields for an event with TSQL and the event name, which is handy because there isn’t a docs page for the properties of each Extended Event. Or maybe it’s why there isn’t a docs page? The software is supposed to be self-documenting, I guess.

SELECT
 oc.name,
 oc.description,
 oc.column_type
FROM sys.dm_xe_object_columns AS oc
WHERE oc.object_name = N'rpc_completed'
AND oc.column_type = N'data'
ORDER BY
 oc.name;
GO

Useful data fields for rpc_completed include the following, along with their descriptions from SQL Server 2025. Earlier versions may return different time increments, SQL Server used milliseconds in its early versions of XEvents:

statement: The text of the statement that was run by the remote procedure call.
cpu_time: The CPU time (in microseconds) used by the remote procedure call.
duration: The time (in microseconds) that the remote procedure call took to be completed.
logical_reads: The number of logical page reads that were issued by the remote procedure call.
physical_reads: The number of physical page reads that were issued by the remote procedure call.
writes: The number of page writes that were issued by the remote procedure call.
row_count: The number of rows that were returned by the remote procedure call.
result: The return value from the remote procedure call.

These fields are automatically captured. Logical_reads, physical_reads, and writes are returned in the count of 8KB pages, so multiply by 8, then divide by 1024. if you want to translate into MB read. Physical_reads is always a subset of logical_reads, because pages need to be pulled into memory to be read.

event_time is also automatically available. This will always be in the UTC time zone, even if you have the timezone for the SQL Server Instance set to something else. I always say that East Coast time is best coast time, but I guess someone got tired of dealing with Daylight Savings Time, and who can blame them.

How to Query Available Actions

To get additional context like the database name or client application name, add those as actions in the EVENT definition.

This query returns actions. There’s no way I’ve found to query the actions that will return meaningful data for a specific Extended Event, it’s a game of trial and error.

SELECT
 package_name = p.name,
 action_name = o.name,
 o.description,
 o.capabilities_desc
FROM sys.dm_xe_objects AS o
JOIN sys.dm_xe_packages AS p
 ON p.guid = o.package_guid
WHERE o.object_type = N'action'
ORDER BY p.name, o.name;

I see 83 actions on SQL Server 2025. Popular actions with their description are:

sqlserver.client_app_name: Collect client application name
sqlserver.database_id: Collect database ID
sqlserver.database_name: Collect current database name
sqlserver.username: Collect username

The following actions are often useful, but don’t return meaningful data for rpc_completed:

sqlserver.query_hash: Collect query hash. Use this to identify queries with similar logic. You can use the query hash to determine the aggregate resource usage for queries that differ only by literal values
sqlserver.query_plan_hash: Collect query plan hash. Use this to identify similar query execution plans. You can use query plan hash to find the cumulative cost of queries with similar execution plans

For RPC_Completed, the information we need is on the statement data field, so I do not collect this action in the scripts below:

sqlserver.sql_text: Collect SQL text

If you are using Resource Governor, you can collect or filter by:

sqlserver.session_resource_group_id: Collect current session resource group ID
sqlserver.session_resource_pool_id: Collect current session resource pool ID

What Sends RPC_Completed Events

rpc_completed events fire for:

Stored procedure calls: When an application or client executes a stored procedure (using the RPC protocol).
Parameterized queries: When using sp_executesql with parameters. You’ll take my dynamic SQL from my cold, dead hands. Also commonly used by .NET applications.
Prepared statement execution: When applications use prepared statements via sp_prepare/sp_execute (more common with ODBC and JDBC than .NET, which typically uses sp_executesql for parameterized queries)

🔥 Common error: Queries you run from a SQL Server Management Studio session do not fire rpc_completed events. These are run as T-SQL batches, even if you run a stored procedure in the T-SQL batch.

For this reason, we'll use PowerShell to test firing RPC events in the code below.

Create an Extended Events Session for RPC_Completed with Minimal Filters

Let’s create a basic Extended Events session to capture all RPC calls.

Even in this simple example, I’m filtering out system database and system session calls as a sensible default. (If you need those, you can delete that whole WHERE clause.)

CREATE EVENT SESSION [capture_rpc_calls]
ON SERVER
 ADD EVENT sqlserver.rpc_completed
 (
 ACTION
 (
 sqlserver.client_app_name,
 sqlserver.database_name,
 sqlserver.username
 )
 WHERE
 (
 [package0].[greater_than_uint64]([sqlserver].[database_id], (4)) /* Exclude master, model, msdb, tempdb */
 AND [package0].[equal_boolean]([sqlserver].[is_system], (0)) /* Exclude system sessions */
 )
 )
 ADD TARGET package0.event_file
 (
 SET filename = N'C:\Temp\capture_rpc_calls.xel',
 max_file_size = 50,
 max_rollover_files = 5
 )
WITH
(
 MAX_MEMORY=4096 KB,
 MAX_DISPATCH_LATENCY=30 SECONDS,
 EVENT_RETENTION_MODE=ALLOW_MULTIPLE_EVENT_LOSS
);
GO

ALTER EVENT SESSION [capture_rpc_calls]
ON SERVER
STATE = START;
GO

Create a Stored Procedure to Test

Let’s also create a stored procedure in SSMS. I’m using the StackOverflow2013 sample database.

USE StackOverflow2013;
GO

/* Create a simple test procedure */
CREATE OR ALTER PROCEDURE dbo.TestProc
 @UserId integer,
 @StartDate datetime
AS
BEGIN
 SELECT
 PostCount = COUNT(*)
 FROM dbo.Posts AS p
 WHERE p.OwnerUserId = @UserId
 AND p.CreationDate >= @StartDate;
END;
GO

Generate RPC_Completed Extended Events with a PowerShell Script

With those in place, let’s generate some rpc_completed events, and also run a query that should not generate an rpc_completed event.

Save this PowerShell script as Test-RpcCompletedEvents.ps1:

# Test script to generate RPC completed events for Extended Events testing
param(
 [string]$ServerInstance = "localhost\FABRIC",
 [string]$Database = "StackOverflow2013"
)

Write-Host "Connecting to $ServerInstance, database $Database..." -ForegroundColor Cyan

$connection = New-Object System.Data.SqlClient.SqlConnection
$connection.ConnectionString = "Server=$ServerInstance;Database=$Database;Integrated Security=True;"

try {
 $connection.Open()
 Write-Host "Connected successfully." -ForegroundColor Green

 # This stored procedure call generates an rpc_completed event
 Write-Host "`n1. Executing stored procedure (generates rpc_completed event)..." -ForegroundColor Yellow
 $command = $connection.CreateCommand()
 $command.CommandType = [System.Data.CommandType]::StoredProcedure
 $command.CommandText = "dbo.TestProc"
 $command.Parameters.AddWithValue("@UserId", 22656) | Out-Null
 $command.Parameters.AddWithValue("@StartDate", "2013-01-01") | Out-Null
 $command.ExecuteNonQuery()
 Write-Host " Stored procedure executed." -ForegroundColor Green

 # This also generates an rpc_completed event (sp_executesql)
 Write-Host "`n2. Executing parameterized query (generates rpc_completed event via sp_executesql)..." -ForegroundColor Yellow
 $command2 = $connection.CreateCommand()
 $command2.CommandType = [System.Data.CommandType]::Text
 $command2.CommandText = "SELECT PostCount = COUNT(*) FROM dbo.Posts AS p WHERE p.OwnerUserId = @UserId AND p.CreationDate >= @StartDate"
 $command2.Parameters.AddWithValue("@UserId", 22656) | Out-Null
 $command2.Parameters.AddWithValue("@StartDate", "2013-01-01") | Out-Null
 $command2.ExecuteNonQuery()
 Write-Host " Parameterized query executed." -ForegroundColor Green

 # This does NOT generate an rpc_completed event (T-SQL batch, not RPC)
 Write-Host "`n3. Executing T-SQL batch (does NOT generate rpc_completed event)..." -ForegroundColor Yellow
 $command3 = $connection.CreateCommand()
 $command3.CommandType = [System.Data.CommandType]::Text
 $command3.CommandText = "SELECT YouDoNotSeeMeInYourTrace = COUNT(*) FROM dbo.Posts AS p WHERE p.OwnerUserId = 22656 AND p.CreationDate >= '2013-01-01'"
 $command3.ExecuteNonQuery()
 Write-Host " T-SQL batch executed." -ForegroundColor Green

 # Additional test: sp_executesql with Users table (for the dynamic SQL filter example)
 Write-Host "`n4. Executing parameterized query against Users table (for dynamic SQL filter testing)..." -ForegroundColor Yellow
 $command4 = $connection.CreateCommand()
 $command4.CommandType = [System.Data.CommandType]::Text
 $command4.CommandText = "SELECT u.Id, u.DisplayName, u.Reputation FROM dbo.Users AS u WHERE u.Location = @Location"
 $command4.Parameters.AddWithValue("@Location", "New York, NY") | Out-Null
 $command4.ExecuteNonQuery()
 Write-Host " Users query executed." -ForegroundColor Green

 Write-Host "`nAll test queries completed!" -ForegroundColor Cyan
 Write-Host "Note: Extended Events may take up to 30 seconds to flush events to file (MAX_DISPATCH_LATENCY)." -ForegroundColor Yellow
}
catch {
 Write-Host "Error: $_" -ForegroundColor Red
 throw
}
finally {
 if ($connection.State -eq 'Open') {
 $connection.Close()
 Write-Host "`nConnection closed." -ForegroundColor Cyan
 }
}

The script executes:

A stored procedure call (generates rpc_completed event)
A parameterized query via sp_executesql (generates rpc_completed event)
A T-SQL batch without parameters (does NOT generate rpc_completed event)
An additional parameterized query against the Users table (for testing dynamic SQL filters)

Run it with:

# With default values (localhost\FABRIC and StackOverflow2013):
.\Test-RpcCompletedEvents.ps1

# Or specify your server and database:
.\Test-RpcCompletedEvents.ps1 -ServerInstance "YourServer\YourInstance" -Database "YourDatabase"

Query the XEvents Trace File and Parse the XML

Read from the trace file to verify what was captured. We set MAX_DISPATCH_LATENCY to 30 seconds in our XEvents trace definition (which is also the default), so it may take up that amount of time for data to be flushed to the file. (Docs on CREATE EVENT SESSION)

/* Read and parse the events from the file */
SELECT
 event_time = event_data.value('(event/@timestamp)[1]', 'datetime2'),
 database_name = event_data.value('(event/action[@name="database_name"]/value)[1]', 'nvarchar(128)'),
 username = event_data.value('(event/action[@name="username"]/value)[1]', 'nvarchar(128)'),
 object_name = event_data.value('(event/data[@name="object_name"]/value)[1]', 'nvarchar(128)'),
 statement = event_data.value('(event/data[@name="statement"]/value)[1]', 'nvarchar(max)'),
 duration_ms = event_data.value('(event/data[@name="duration"]/value)[1]', 'bigint') / 1000,
 cpu_time_ms = event_data.value('(event/data[@name="cpu_time"]/value)[1]', 'bigint') / 1000,
 logical_reads = event_data.value('(event/data[@name="logical_reads"]/value)[1]', 'bigint'),
 row_count = event_data.value('(event/data[@name="row_count"]/value)[1]', 'bigint'),
 full_event_xml = event_data
FROM
(
 SELECT
 event_data = CAST(event_data AS xml)
 FROM sys.fn_xe_file_target_read_file
 (
 N'C:\Temp\capture_rpc_calls*.xel',
 NULL,
 NULL,
 NULL
 )
) AS xe;
GO

The results show:

Stored procedure executions were captured, including sp_executesql calls (RPC calls)
The ad-hoc SELECT was not (language event)

The statements for calls to sp_executesql were captured with the parameter values. For example:

exec sp_executesql N'SELECT PostCount = COUNT(*) FROM dbo.Posts AS p WHERE p.OwnerUserId = @UserId AND p.CreationDate >= @StartDate',N'@UserId int,@StartDate nvarchar(10)',@UserId=22656,@StartDate=N'2013-01-01'

Filter RPC_Completed by Object Name

You usually don’t want to capture every RPC call. Filtering to just capture what you need makes the trace lighter, and also your life easier going through the trace when problem solving.

Filtering by object name and by the login executing the commands is very useful:

CREATE EVENT SESSION [capture_specific_proc]
ON SERVER
 ADD EVENT sqlserver.rpc_completed
 (
 ACTION
 (
 sqlserver.client_app_name,
 sqlserver.database_name,
 sqlserver.username
 )
 WHERE
 (
 [object_name] = N'TestProc'
 AND [package0].[greater_than_uint64]([sqlserver].[database_id], (4)) /* Exclude master, model, msdb, tempdb */
 AND [package0].[equal_boolean]([sqlserver].[is_system], (0)) /* Exclude system sessions */
 /* Uncomment to filter by SQL login */
 /* AND [sqlserver].[username] = N'SQLLoginName' */
 /* Uncomment to filter by Windows login */
 /* AND [sqlserver].[username] = N'DOMAIN\WindowsUser' */
 )
 )
 ADD TARGET package0.event_file
 (
 SET filename = N'C:\Temp\capture_specific_proc.xel',
 max_file_size = 50,
 max_rollover_files = 5
 )
 WITH
 (
 MAX_MEMORY=4096 KB,
 MAX_DISPATCH_LATENCY=30 SECONDS,
 EVENT_RETENTION_MODE=ALLOW_MULTIPLE_EVENT_LOSS
 );
GO

ALTER EVENT SESSION [capture_specific_proc]
ON SERVER
STATE = START;
GO

The object_name predicate matches the procedure name, but without the schema.

The example above uses an equality. To pattern match with a case-insensitive comparison, use the like_i_sql_unicode_string predicate operator instead:

[sqlserver].[like_i_sql_unicode_string]([object_name], N'test%')

Rerunning Test-RpcCompletedEvents.ps1 and then the query to read from the file (adapted for this trace file name), this captured only the call to TestProc:

Filter RPC_Completed by Object Name and SQL Text

You can combine object_name and statement filters for more precise targeting. This is useful when you want to ensure you’re capturing a specific procedure and also verify it’s being called in a particular way.

This is also useful because there is not a schema_name field to filter on.

The following example only works if the exact statement being issued leads with exec dbo.TestProc.

CREATE EVENT SESSION [capture_testproc_with_sqltext]
ON SERVER
 ADD EVENT sqlserver.rpc_completed
 (
 ACTION
 (
 sqlserver.client_app_name,
 sqlserver.database_name,
 sqlserver.username
 )
 WHERE
 (
 [object_name] = N'TestProc'
 AND [sqlserver].[like_i_sql_unicode_string]([statement], N'exec dbo.TestProc%')
 AND [package0].[greater_than_uint64]([sqlserver].[database_id], (4)) /* Exclude master, model, msdb, tempdb */
 AND [package0].[equal_boolean]([sqlserver].[is_system], (0)) /* Exclude system sessions */
 /* Uncomment to filter by SQL login */
 /* AND [sqlserver].[username] = N'SQLLoginName' */
 /* Uncomment to filter by Windows login */
 /* AND [sqlserver].[username] = N'DOMAIN\WindowsUser' */
 )
 )
 ADD TARGET package0.event_file
 (
 SET filename = N'C:\Temp\capture_testproc_with_sqltext.xel',
 max_file_size = 50,
 max_rollover_files = 5
 )
 WITH
 (
 MAX_MEMORY=4096 KB,
 MAX_DISPATCH_LATENCY=30 SECONDS,
 EVENT_RETENTION_MODE=ALLOW_MULTIPLE_EVENT_LOSS
 );
GO

ALTER EVENT SESSION [capture_testproc_with_sqltext]
ON SERVER
STATE = START;
GO

I tested this to make sure it works, but the output is the same as above: it captures dbo.TestProc.

Filter RPC_Completed for a Specific Query Called by sp_executesql

It’s common for applications to use sp_executesql to execute parameterized queries against SQL Server.

In this case, the object_name will be sp_executesql. To filter for specific dynamic SQL, you need to filter on the statement data field, which contains the actual SQL statement being executed.

CREATE EVENT SESSION [capture_specific_dynamic_sql]
ON SERVER
 ADD EVENT sqlserver.rpc_completed
 (
 ACTION
 (
 sqlserver.client_app_name,
 sqlserver.database_name,
 sqlserver.username
 )
 WHERE
 (
 [object_name] = N'sp_executesql'
 AND [sqlserver].[like_i_sql_unicode_string]([statement], N'%FROM dbo.Users%')
 AND [package0].[greater_than_uint64]([sqlserver].[database_id], (4)) /* Exclude master, model, msdb, tempdb */
 AND [package0].[equal_boolean]([sqlserver].[is_system], (0)) /* Exclude system sessions */
 /* Uncomment to filter by SQL login */
 /* AND [sqlserver].[username] = N'SQLLoginName' */
 /* Uncomment to filter by Windows login */
 /* AND [sqlserver].[username] = N'DOMAIN\WindowsUser' */
 )
 )
 ADD TARGET package0.event_file
 (
 SET filename = N'C:\Temp\capture_specific_dynamic_sql.xel',
 max_file_size = 50,
 max_rollover_files = 5
 )
 WITH
 (
 MAX_MEMORY=4096 KB,
 MAX_DISPATCH_LATENCY=30 SECONDS,
 EVENT_RETENTION_MODE=ALLOW_MULTIPLE_EVENT_LOSS
 );
GO

ALTER EVENT SESSION [capture_specific_dynamic_sql]
ON SERVER
STATE = START;
GO

It’s a match!

Filter RPC_Completed by Duration or Logical Reads

You may want to only capture queries of a certain duration or that do more than a given amount of IO.

CREATE EVENT SESSION [capture_by_duration_reads]
ON SERVER
 ADD EVENT sqlserver.rpc_completed
 (
 ACTION
 (
 sqlserver.client_app_name,
 sqlserver.database_name,
 sqlserver.username
 )
 WHERE
 (
 [sqlserver].[database_name] = N'StackOverflow2013'
 AND (
 [object_name] = N'TestProc'
 OR [object_name] = N'UpdateUserReputation'
 OR [object_name] = N'GetTopUsers'
 )
 AND (
 [duration] > 5000000 /* More than 5 seconds (in microseconds) */
 OR [logical_reads] > 10000 /* More than 10,000 logical reads */
 )
 AND [package0].[greater_than_uint64]([sqlserver].[database_id], (4)) /* Exclude master, model, msdb, tempdb */
 AND [package0].[equal_boolean]([sqlserver].[is_system], (0)) /* Exclude system sessions */
 /* Uncomment to filter by SQL login */
 /* AND [sqlserver].[username] = N'SQLLoginName' */
 /* Uncomment to filter by Windows login */
 /* AND [sqlserver].[username] = N'DOMAIN\WindowsUser' */
 )
 )
 ADD TARGET package0.event_file
 (
 SET filename = N'C:\Temp\capture_by_duration_reads.xel',
 max_file_size = 50,
 max_rollover_files = 5
 )
 WITH
 (
 MAX_MEMORY=4096 KB,
 MAX_DISPATCH_LATENCY=30 SECONDS,
 EVENT_RETENTION_MODE=ALLOW_MULTIPLE_EVENT_LOSS
 );
GO

ALTER EVENT SESSION [capture_by_duration_reads]
ON SERVER
STATE = START;
GO

This session will only capture calls that meet your thresholds. One event matches the criteria:

Extended Events Session Cleanup Script

This stops and drops all Extended Events sessions created in this post (and any others that begin with ‘capture’)

/* Stop and drop all sessions named like capture% */
DECLARE
 @session_name nvarchar(128);

DECLARE session_cursor CURSOR LOCAL FAST_FORWARD FOR
 SELECT
 ses.name
 FROM sys.server_event_sessions AS ses
 WHERE ses.name like N'capture%';

OPEN session_cursor;

FETCH NEXT FROM session_cursor
INTO @session_name;

WHILE @@FETCH_STATUS = 0
BEGIN
 /* Stop the session if it's running */
 DECLARE
 @stop_sql nvarchar(max) = N'
 ALTER EVENT SESSION ' + QUOTENAME(@session_name) + N'
 ON SERVER
 STATE = STOP;';

 BEGIN TRY
 EXECUTE sp_executesql @stop_sql;
 END TRY
 BEGIN CATCH
 /* Session might not be running, continue */
 END CATCH;

 /* Drop the session */
 DECLARE
 @drop_sql nvarchar(max) = N'
 DROP EVENT SESSION ' + QUOTENAME(@session_name) + N'
 ON SERVER;';

 EXECUTE sp_executesql @drop_sql;

 FETCH NEXT FROM session_cursor
 INTO @session_name;
END;

CLOSE session_cursor;
DEALLOCATE session_cursor;
GO

/* Verify all sessions are removed */
SELECT
 ses.name
FROM sys.server_event_sessions AS ses
WHERE ses.name LIKE N'capture%';
GO

Rpc_completed Filtering Strategies

When filtering rpc_completed events…

Start specific:
- Filtering by the [sqlserver].[username] running the query is usually a good thing
- Filter by object_name if troubleshooting a known procedure
  - Equality example: [object_name] = N'TestProc'
  - Like example: [sqlserver].[like_i_sql_unicode_string]([object_name], N'test%')
- For parameterized queries sent from applications (like Entity Framework), the object_name will usually be sp_executesql, so filter by [object_name] = N'sp_executesql' and add filters on the statement field to narrow down to specific queries
- Filter on database_name to avoid capturing the same procedure name in different databases
  - Equality example: [sqlserver].[database_name] = N'StackOverflow2013'
- Otherwise at least filter out system databases if you can with [package0].[greater_than_uint64]([sqlserver].[database_id], (4))
- Filter out system activity with [package0].[equal_boolean]([sqlserver].[is_system], (0))
You can filter on schema name, but only in the statement: There is no schema name field to filter on for rpc_completed events. If you need to filter by schema, you can filter on the statement field using [sqlserver].[like_i_sql_unicode_string]([statement], N'exec dbo.TestProc%') to match schema-qualified procedure calls. This approach depends on how the application calls the procedure, so you’ll need to take samples first to narrow this down.
Combine object and performance filters: Filter by object name AND duration to catch only the slow executions for RPC events. You can also filter by logical_reads to exclude calls that do a minimum of IO. For parameterized queries from applications, combine [object_name] = N'sp_executesql' with performance filters like [duration] > 5000000 to capture expensive parameterized queries without capturing every single execution.
Test filters in development first: Create the session in a dev environment to verify your filters capture what you expect before deploying to production. Remember that queries run from SSMS do not fire rpc_completed events (they’re T-SQL batches, not RPC calls), so use the PowerShell script or an application connection to test your filters.
Sample by session_id for high-volume scenarios: You can sample a percentage of sessions using the divides_by_uint64 function on session_id. The example sampling traces in the XEvents tooling suggest this technique:

WHERE (
 [package0].[divides_by_uint64]([sqlserver].[session_id], (5)) /* Sample 20% of sessions (every 5th session) */
)

Adjust the divisor in divides_by_uint64 to change the sampling rate: use 10 for 10% sampling, 20 for 5% sampling, etc.

Capture the Signal, Not the Noise

Without filtering, XEvents traces often generate an overwhelming amount of data.

The filtering techniques in this post (object name, SQL text matching, performance thresholds, and session sampling) focus on the events that matter. Filtering by specific procedures, expensive queries, or a sample of sessions captures the signal needed for troubleshooting without extraneous noise.

Erik Darling and Kendra Little Talk AI, Databases, and SQL Server 2025

help@KendraLittle.com (Kendra Little) — Tue, 09 Dec 2025 00:00:00 +0000

Erik Darling joins me on the Dear SQL DBA Podcast to chat about AI tools, why they work better for Python and PowerShell than SQL, and what’s exciting (and what’s not) in SQL Server 2025.

Along the way we describe the ONE THING we always want to configure in Resource Governor, why at least one SQL Server 2025 feature should be backported to 2022, and whether the universe is text files all the way down.

Audio only

Video

In this Conversation

Erik and I talked about how we currently use AI tools, plus topics related to SQL Server 2025 in this episode.

Erik’s approach to AI is to treat it like a junior developer rather than a SQL Server expert. He uses AI primarily to automate tasks. AI helps him working with APIs to help him reduce manual work to run his business.

We discussed why AI produces better code for Python and PowerShell than SQL. Scripting languages have less ambiguity and clearer error messages, while SQL requires deep context about indexes, data distribution, and performance that AI struggles with.

The highlights we discussed with SQL Server 2025 are Query Store on readable secondaries (finally), Resource Governor now available in Standard Edition, and Standard Edition supporting up to 256GB of memory. The AI features are limited, with DISKANN indexes (vector indexes) making tables read-only. We discussed being wary of enabling preview features with a database scoped level flag: seems fine for development, but this could have unwelcome surprises if you do this in production.

Where to Jump In

Here’s where the major topics start in the conversation:

[0:00] Introduction and opening banter about AI tools in interviews
[2:42] Erik’s approach to using AI tools and agents for productivity
[11:06] Why AI works better for Python and PowerShell than SQL
[19:45] SQL Server 2025: Finally, Query Store on readable secondaries
[27:20] Standard Edition improvements: Resource Governor and memory limits
[31:50] AI features in SQL Server 2025 and preview features concerns

More Dear SQL DBA

Check out the episode index for more conversations about database problems, performance tuning, and the real challenges database professionals face.

Azure SQL Managed Instance Memory-to-Core Math Still Doesn't Work, Even in GPv2

help@KendraLittle.com (Kendra Little) — Mon, 08 Dec 2025 00:00:00 +0000

Microsoft recently announced that Azure SQL Managed Instance Next-gen General Purpose (GPv2) is generally available. GPv2 brings significant storage performance improvements over GPv1. If you’re using GPv1, you should plan to upgrade.

But GPv2 still has the same memory-to-core ratio problem that makes Managed Instance a rough deal for running SQL Server. SQL Server is engineered to use lots of memory—it’s a rare OLTP or mixed-OLTP workload that doesn’t need significant cache for reliable performance. We’ll have a look at the pricing math.

What’s New in GPv2

GPv2 entered public preview in March 2024 and remained there for 20 months before reaching general availability. The new tier brings significant improvements over GPv1. According to the announcement:

Better I/O latency: Average latency drops from 5-10ms to 3-4ms
Higher IOPS: Maximum data IOPS increases from 30-50k to 80k
More storage: Maximum storage increases from 16TB to 32TB
More databases per instance: Up to 500 databases per instance (up from 100)
More vCores: Maximum vCores increases from 80 to 128
Memory slider: New flexible memory configuration options
Faster scaling: Storage and IOPS can be resized independently of compute

🔥 Note on Memory Slider: The maximum amount of memory per vCore has not increased. Azure SQL Managed Instance still maxes out at 13.6 GB per vCore for memory-optimized hardware.

Why GPv1 is a Problem

If you’ve been using Azure SQL Managed Instance General Purpose, you’ve likely experienced storage pains with GPv1. I’ve written about how Azure SQL Managed Instance storage regularly stalls for 15-60 seconds in GPv1, despite Microsoft’s documentation claiming “5-10 ms” latency.

These storage stalls cause real problems. SQL Server treats I/O requests that take longer than 15 seconds as serious errors because these stalls cause blocking, query timeouts, and cascading performance issues that last far longer than the initial stall.

GPv1 uses Azure Blob Storage for database files, which has fundamental limitations, including file size-based IOPS throttling that causes customers to have to grow files unnecessarily and makes performance management difficult and unpredictable. GPv2 uses Azure Elastic SAN for storage.

Why Features Exiting Preview Matters

Preview features come with important caveats.

Now that GPv2 is generally available, it’s backed by Microsoft’s standard SLAs. If there are outages or problems, you’re entitled to service credits. The feature is committed to long-term support, and Microsoft can’t just remove it overnight.

All Managed Instance Customers Should Consider GPv2

If you’re running Azure SQL Managed Instance GPv1, plan your upgrade to GPv2. The performance improvements are significant, and the storage architecture is fundamentally better.

It’s worth your time to plan this proactively. The November 2022 feature wave was announced in November 2022 and I’m pretty sure it didn’t finish rolling out until sometime in 2024. If you aren’t proactive about upgrades, you could be waiting a while based on that past history.

Microsoft notes that zonal redundancy isn’t included in the initial GA release but is being worked on.

If you’re using Azure SQL Managed Instance Business Critical, consider if General Purpose might meet your needs with GPv2. You may have to overprovision vCores to get the memory you need for your workload, but the pricing difference might still make sense if the performance works. See the pricing comparisons below to note the differences between Business Critical and General Purpose pricing.

You Should Still Monitor for Storage Latency Issues

Even though GPv2 uses better storage technology, you should still monitor your SQL Server error logs for storage latency errors. The underlying causes may be different from GPv1’s problems, but storage latency issues can still occur for various reasons: network problems, Azure infrastructure issues, or other factors.

I’ve written about how to find these errors in the SQL Server log. Look for messages containing “longer than 15 seconds” in your error log. Use Erik Darling’s free sp_loghunter utility to search the logs programmatically.

On GPv1, these storage latency problems went largely unresolved and undiscussed for many years. Don’t let that happen again. If you see storage latency issues on GPv2, document them and report them to Microsoft support.

Microsoft Still Needs to Provide More for Managed Instance Customers

While GPv2 is a welcome improvement, Microsoft still has work to do to compete effectively with other cloud database offerings. I’ve written about three reasons RDS SQL Server is better than Azure SQL Managed Instance, and the memory-to-core ratio issue is particularly important.

SQL Server OLTP workloads benefit enormously from having large amounts of memory to reduce physical I/O. Memory access is always dramatically faster than storage access, and SQL Server is engineered to maximize use of the buffer pool and other caches.

It’s more critical than ever that Microsoft improves their memory-to-core ratios to provide a platform suitable for cache-hungry SQL Server. The comparison is stark between SQL Server 2025 prices for Standard Edition and pricing for Managed Instance General Purpose.

SQL Server 2025 Pricing

Platform	Cores	Total Memory	Memory per Core	Cost per Month	Cost per Year
SQL Server 2025 Standard (one-time license)	4	256 GB (max for Standard)	64 GB	$657.50	$7,890
SQL Server 2025 Enterprise (one-time license)	4	256 GB (can go much higher without increasing licensing cost)	64 GB	$2,520.50	$30,246

Pricing source: SQL Server 2025 pricing, assumes 4 cores (2-core packs × 2)
Price doesn’t include hardware, staff to manage Availability Groups, or Software Assurance.

What would I need to pay to get the same amount of memory on Azure SQL Managed Instance? I’m just estimating 256 GB of memory here, not a ton for a production database. But 13.6 GB of memory per core is still the maximum available in either General Purpose or Business Critical (you must choose Memory Optimized premium-series hardware to get it).

Managed Instance Pricing to Get the Same Amount of Memory

You’d have to pay for 19 vCores to get 256GB or memory on Managed Instance. 19 vCores isn’t an option, so you’ll need to pay for 20 vCores. Here’s what the Azure Pricing calculator says for East US as of November 30, 2025:

Platform	Cores	Total Memory	Memory per Core	Cost per Month	Cost per Year
Managed Instance GPv2 (Next Generation General Purpose)	20	272 GB	13.6 GB	$5,051.10	$60,613.20
Managed Instance Business Critical	20	272 GB	13.6 GB	$12,672.80	$152,073.60

These prices vary by region. In Central US your budget needs to be bigger:

Platform	Cores	Total Memory	Memory per Core	Cost per Month	Cost per Year
Managed Instance GPv2 (Next Generation General Purpose)	20	272 GB	13.6 GB	$5,883.30	$70,599.60
Managed Instance Business Critical	20	272 GB	13.6 GB	$14,322.60	$171,871.20

Pricing source: Azure pricing calculator, Pay as You Go, US East region, 4 vCores Next Generation General Purpose, Premium-Series memory optimized (Hardware Type), 32GB storage minimum, no zone redundancy

These estimates only describe 4 cores for the one-time license workload– if you’re reading this article you’ve got way more cores than that. Is the extra recurring cost worth not buying hardware (or using a VM in the cloud where you limit the active cores you have to license) or hiring someone to manage Availability Groups? ¯\_(ツ)_/¯

In both cases, it’s possible to get better deals on pricing: either by buying from resellers or package deals in the case of on-prem, or by reserved instances and Azure credits deals in the case of cloud services.

🔥 Why am I so obsessed with memory? SQL Server has been built to leverage caching for performance. The storage speeds in GPv2 and the local SSD options for Business Critical in Azure SQL ain't that great when it comes to modern storage speeds. If you need reliable database performance, sufficient memory to reduce the amount of physical I/O you do on these systems is table stakes.

I do believe there are very strong features in Azure SQL Managed Instance and that it can be a compelling product, but it badly needs more memory per vCore. This is a major flaw that undercuts the engineering achievements of the service.

Y’all, I don’t particularly enjoy managing Windows Clusters and Availability Groups, personally. I love managed database services in the cloud. But Azure SQL Managed Instance needs to support memory configurations suitable to the product that it runs at a more reasonable price. Changes like that are what really makes SaaS database services take off.

Index Bloat in Postgres: Why It Matters, How to Identify, How to Resolve

help@KendraLittle.com (Kendra Little) — Mon, 01 Dec 2025 05:00:00 -0500

Index bloat in Postgres can cause problems, but it’s easy to miss.

I’ve written about how vacuum problems can prevent PostgreSQL from using covering indexes, and index bloat is one of the things that can make vacuum struggle.

Here’s what you need to know about index bloat, how to find it, and how to fix it.

What Is Index Bloat?

“Index bloat” describes an index that takes up significantly more space on disk than it actually needs. This occurs because Postgres stores indexes in 8 KB fixed-size pages which can accumulate empty space over time.

When you delete or update rows, the old index entries become “dead tuples”—they’re no longer valid, but they remain on the page taking up space. Vacuum marks that space as available for reuse, but it doesn’t compact the pages themselves. You end up with pages that have a lot of empty space scattered throughout them.

🔥 VAC FACT: VACUUM FULL can remove bloat, but it requires heavy locks and rewrites the whole table, so this isn't a workable option for databases unless there is a regular long downtime period and you don't mind the IO.

How Index Bloat Happens

Index bloat accumulates through normal database operations:

Deletes: When you delete rows, the index entries pointing to those rows become dead tuples, but they stay in the index until vacuum runs
Updates: Postgres implements updates as delete-plus-insert under MVCC (Multi-Version Concurrency Control). The old index entry becomes dead, and Postgres creates a new one
Page splits: When a B-tree index page becomes full, PostgreSQL splits it into two pages.
Low fillfactor: You can reduce page splits by setting a lower fillfactor when creating indexes, which leaves more free space on each page. B-tree indexes default to 90% fillfactor. Indexes with lower fillfactor leave more empty space. This isn’t the same thing as bloat (it’s on purpose), but going too low can have similar impacts.
High update/delete activity: Tables with frequent updates or deletes accumulate bloat faster

To reiterate what’s mentioned above (it’s a common misunderstanding): autovacuum removes dead tuples from tables and marks space in indexes as available for reuse, but it doesn’t reclaim space in indexes.

Why Index Bloat Causes Problems

When you have significant index bloat, vacuum has to work harder. Vacuum needs to scan indexes to find and remove references to dead tuples. If your indexes are bloated, they contain more pages because the pages aren’t compacted—an index that should be 10 GB might be 15 GB or more instead if bloat gets bad, and imagine how this expands for larger indexes. Vacuum has to scan through all the extra pages, which takes more time and consumes more CPU and IO resources. The more bloat, the more pages to scan, and the longer vacuum takes.

In systems with heavy write workloads, this can create a vicious cycle: vacuum runs slowly because of bloat, which means it can’t keep up with new dead tuples, which creates more bloat, etc.

I’ve seen cases where vacuum was taking hours on tables that should have been much quicker to vacuum, all because the indexes were severely bloated.

How to Identify Index Bloat

AWS provides a useful script for calculating index bloat percentage in RDS PostgreSQL instances: Rebuilding indexes. This script calculates bloat_pct for each index and helps you identify which indexes need attention.

The script compares the actual index size to the expected size based on the number of tuples, giving you a percentage of bloat. AWS recommends running REINDEX when the bloat percentage is greater than 20%, but let your conscience be your guide.

How to Fix Index Bloat

Postgres supports online index rebuilds using REINDEX CONCURRENTLY. Under concurrent mode, the index remains available for reads and writes during the rebuild operation. The PostgreSQL documentation explains that REINDEX CONCURRENTLY builds a new index alongside the existing one, performs multiple passes to capture changes made during the rebuild, then switches constraints and names to point to the new index before dropping the old one. This process takes longer than a standard rebuild but allows normal operations to continue throughout.

Here’s the syntax:

REINDEX INDEX CONCURRENTLY index_name;

Or for all indexes on a table:

REINDEX TABLE CONCURRENTLY table_name;

The CONCURRENTLY keyword makes it online. Without it, REINDEX takes an exclusive lock on the table, blocking all access.

A Gotcha: Failed Concurrent Rebuilds Leave Invalid Indexes

If a REINDEX CONCURRENTLY operation fails partway through, Postgres leaves behind an “invalid” index that you can’t use. According to the PostgreSQL documentation, when REINDEX CONCURRENTLY fails, Postgres leaves behind a temporary index with a name ending in _ccnew. This index won’t function for reads (but still has overhead for updates), so drop it before trying the rebuild again.

You can find invalid indexes by looking for indexes whose name ends in %_ccnew:

SELECT
 schemaname,
 indexname,
 indexdef
FROM pg_indexes
WHERE schemaname = 'public'
/* Concurrent rebuilds create temporary indexes named like this */
AND (
 indexname LIKE '%\_ccnew'
 OR indexname NOT IN (
 SELECT indexrelname
 FROM pg_stat_user_indexes
 )
);

Or query the pg_index, pg_class, and pg_namespace system catalogs to find indexes marked as invalid:

SELECT
 n.nspname AS schema_name,
 c.relname AS index_name,
 pg_size_pretty(pg_relation_size(c.oid)) AS index_size
FROM pg_index i
JOIN pg_class c ON i.indexrelid = c.oid
JOIN pg_namespace n ON c.relnamespace = n.oid
WHERE NOT i.indisvalid
AND n.nspname = 'public';

If you find invalid indexes from a failed concurrent rebuild, drop them before trying again. You can use DROP INDEX or DROP INDEX CONCURRENTLY:

DROP INDEX CONCURRENTLY schema_name.invalid_index_name;

The CONCURRENTLY keyword avoids blocking by not requiring an ACCESS EXCLUSIVE lock on the table.

Sample Scripts to Reproduce Index Bloat

Here’s a script you can use on a test database to create and observe index bloat.

🔥 Note: This demo creates a small table that you probably wouldn't worry about in a production environment, but it'll give you an idea of how this works. The script requires the pgstattuple extension, and is not as comprehensive and efficient as the script linked to in the How to Identify Index Bloat section above.

CREATE EXTENSION IF NOT EXISTS pgstattuple;


DROP TABLE IF EXISTS test_bloat;

CREATE TABLE test_bloat (
 id integer PRIMARY KEY,
 user_id integer NOT NULL,
 order_date date NOT NULL,
 status_code integer NOT NULL
);

CREATE INDEX idx_test_bloat_multi ON test_bloat (user_id, order_date, status_code);

/* Insert some sample data */
INSERT INTO test_bloat (id, user_id, order_date, status_code)
SELECT
 generate_series(1, 1000000),
 (random() * 10000)::integer,
 CURRENT_DATE - (random() * 365)::integer,
 (random() * 10)::integer;

/* Check index size and empty space on pages */
SELECT
 s.schemaname,
 s.indexrelname AS indexname,
 /* Total size of the index on disk */
 pg_size_pretty(pg_relation_size(s.indexrelid)) AS index_size,
 /* Total number of pages: leaf pages + internal pages + empty pages + deleted pages */
 (stats.leaf_pages + stats.internal_pages + stats.empty_pages + stats.deleted_pages) AS total_pages,
 /* Average density of leaf pages (0-100, lower = more empty space on pages) */
 ROUND(stats.avg_leaf_density::numeric, 1) AS avg_leaf_density,
 /* Empty space on leaf pages based on current density */
 /* If density is 50%, then 50% of leaf page space is empty */
 pg_size_pretty(
 (
 stats.leaf_pages * current_setting('block_size')::bigint *
 (100.0 - stats.avg_leaf_density) / 100.0
 )::bigint
 ) AS empty_space,
 /* Percentage of space on leaf pages that's currently empty */
 ROUND(
 (100.0 - stats.avg_leaf_density)::numeric,
 1
 ) AS empty_space_pct
FROM pg_stat_user_indexes AS s
/* Use pgstatindex() to get actual page-level statistics from the index */
CROSS JOIN LATERAL pgstatindex(s.schemaname || '.' || s.indexrelname) AS stats
WHERE s.relname = 'test_bloat';

Here’s the initial size and density - the primary key pages are 90.1% full, the multi column index pages are 70.1% full.

/* Delete a large portion of rows */
DELETE FROM test_bloat WHERE id % 2 = 0;

/* You can recheck the size using the query above, but nothing will
be different until you run VACUUM. */

/* Run VACUUM ANALYZE */
VACUUM ANALYZE test_bloat;

/* Recheck the size using the query above */

Now here’s our index size and empty space: the primary key pages are 45% full and the multi column index pages are 35.5% full.

/* Rebuild index concurrently to reclaim space */
REINDEX INDEX CONCURRENTLY idx_test_bloat_multi;
REINDEX INDEX CONCURRENTLY test_bloat_pkey;

/* Recheck the size using the query above */

Here’s the size and empty space after rebuilds: each index is 90% full.

/* Clean up */
DROP TABLE test_bloat;

Index Bloat Creeps Up Gradually

Have you checked your indexes for bloat? Have you implemented automated maintenance for bloat?

How to See Rowcounts and Execution Time for In-Flight Queries in SQL Server

help@KendraLittle.com (Kendra Little) — Mon, 24 Nov 2025 00:00:00 +0000

I frequently need to see rowcounts and execution time for queries while they’re running. Maybe I’m troubleshooting a slow query that’s still executing, or I want to understand which operators are causing the slowdown before the query completes.

Last week at the PASS Summit I learned some little nuances about how this works that I’d missed.

Tldr: Quick Method

For application queries: Use sp_WhoIsActive with @get_plans = 1 to see rowcounts for in-flight queries. Rowcounts show up even if the session running the query doesn’t have standard profiling enabled.

When reproducing a slow query from my own session: Enable actual execution plans (Ctrl+M) in the session running the query, then use sp_WhoIsActive with @get_plans = 1 in another session if it runs long. This includes not only rowcounts per operator, but also query execution times per operator in the in-flight plan that you nab.

EXECUTE sp_WhoIsActive
 @get_plans = 1;

If you want to save a script for this, you can use SET STATISTICS XML ON in the script that is running the query you want to profile instead of enabling actual execution plans in that session.

My Misunderstanding

I used to think that execution times per operator showing up in sp_WhoIsActive was random. The rowcounts pretty much always show up on recent versions of SQL Server, but sometimes I’d see execution times per operator, sometimes I wouldn’t. It seemed like luck.

I’d missed the memo that execution times per operator only appear when standard query profiling is enabled in the session running the query. It’s not random at all. I was enabling it without realizing it—when I had already enabled actual execution plans on queries I was running for repro, that automatically enabled standard query profiling.

Standard query profiling enables additional runtime statistics collection. When enabled, SQL Server collects execution times per operator as the query runs. Rowcounts are available even without standard profiling enabled, but execution times per operator require it.

How It Works

sp_WhoIsActive queries the sys.dm_exec_query_statistics_xml dynamic management view, which returns the query execution plan for in-flight requests with transient statistics.

Query this DMV directly if you can’t install sp_WhoIsActive or prefer a simple approach:

SELECT
 eqs.query_plan,
 er.session_id,
 er.request_id,
 er.database_id,
 er.start_time,
 er.[status],
 er.wait_type,
 er.wait_resource,
 cpu_time_sec = (er.cpu_time / 1000.),
 logical_reads_MB = (er.logical_reads * 8) / 1024. / 1024.,
 granted_memory_MB = er.granted_query_memory * 8 / 1024. / 1024.,
 er.dop,
 er.query_hash,
 er.query_plan_hash
FROM sys.dm_exec_requests AS er
CROSS APPLY sys.dm_exec_query_statistics_xml(er.session_id) AS eqs
WHERE er.session_id <> @@SPID;

Let’s Look at This in Action

Here’s a script that takes a bit of time to run, which can be useful for testing this against the StackOverflow2013 sample database. This drops any nonclustered indexes and auto-created stats that exist, then creates two nonclustered indexes:

USE StackOverflow2013;
GO

EXECUTE dbo.DropIndexes;
GO

CREATE INDEX
 Reputation
ON dbo.Users
 (Reputation)
INCLUDE
 (
 Id,
 DisplayName
 )
WITH
 (SORT_IN_TEMPDB = ON, DATA_COMPRESSION = PAGE);
GO

CREATE INDEX
 ParentId_OwnerUserId
ON dbo.Posts
 (
 ParentId,
 OwnerUserId
 )
WITH
 (SORT_IN_TEMPDB = ON, DATA_COMPRESSION = PAGE);
GO

After I get those index creation operations running in one session, I run the query above in a second session to capture the in-flight plan.

I do NOT have actual execution plans enabled on the query that is creating the indexes, so the in-flight plan has rowcounts on each operator, but no timing information:

In this execution plan, you can see rowcounts on each operator showing how many rows have been processed so far:

[Clustered Index Scan]: 16,677,114 rows processed (97% of 17,142,200 estimated), 92% estimated cost
[Sort]: 0 rows processed, 7% estimated cost
[Online Index Insert]: 0 rows processed, 1% estimated cost

If I rerun this process but first run SET STATISTICS XML ON; or enable actual execution plans in the session that is creating the indexes, I additionally see execution time per operator in the in-flight plan:

With standard query profiling enabled, each operator shows both rowcounts and timing information:

[Clustered Index Scan]: 14,901,911 rows processed (86% of 17,142,200 estimated), 11.607 seconds elapsed, 92% estimated cost
[Sort]: 0 rows processed, 13.813 seconds elapsed, 7% estimated cost
[Online Index Insert]: 0 rows processed, 13.813 seconds elapsed, 1% estimated cost

Practical Tips

Use this technique when troubleshooting slow queries or understanding query progress. It’s particularly helpful for identifying which operators are causing slowdowns before a query completes.

Note that cost estimates in the plan (which show as percentages) are estimates made at the time the query was compiled. These cost estimates are NOT updated during or after runtime of the query. Queries may also reuse an execution plan where the cost estimates were made previously for a different query execution, based on the parameters that were sniffed when the plan was compiled.

For more details on the query profiling infrastructure, see Microsoft’s documentation.

Three Reasons RDS SQL Server Is Better Than Azure SQL Managed Instance

help@KendraLittle.com (Kendra Little) — Wed, 08 Oct 2025 14:25:57 -0400

While every managed database service has high points and low points, there are three things that make RDS for SQL Server shine in comparison to Azure SQL Managed Instance: options for a higher memory:vCPU ratio, a well documented API that works beautifully with python, and fast and effective customer support that isn’t painful to use.

It’s just three things, but they make a huge difference. And these are three things that Microsoft can, and SHOULD, really fix for their cloud database services.

SQL Server RDS Offers a High Memory:vCPU Ratio, Which Really Matters for OLTP Workloads

AWS offers a variety of database instance class types for RDS.

The standout among these for SQL Server OLTP workloads is the db.x2iedn instance class.

The db.x2iedn class offers:

32GB:1 ratio of memory to virtual CPU
Local SSD for tempdb

As the documentation says:

The X2 instance family provides the highest memory to vCPU ratio compared to all supported RDS DB instance types. This is ideal for SQL Server workloads that are sensitive to per-core licensing, such as SQL Server Enterprise edition and SQL Standard Edition. Your workload can benefit greatly from the higher memory per vCPU (32GiB:1vCPU) offered by X2iedn and reduce the number of vCPU provisioned.

No lies detected– this is good advice. SQL Server per-core licensing is expensive, and SQL Server OLTP workloads massively benefit from having large amounts of memory to reduce the amount of physical IO. No matter what storage type we are talking about, memory access is always dramatically faster than storage access, and SQL Server is engineered to maximize and optimize the use of the buffer pool and other caches to reduce IO. A SQL Server loves memory like I love making lists: I can live without it, but performance significantly suffers.

One problem with managed SQL Servers in the cloud in general is that memory to core ratios SUCK out there. In Azure, if you use “memory optimized” hardware, you can get up to 13.6GB:1 memory to vCPU ratio max.

If you pay the big bucks for Business Critical, you can get local SSD, but I repeat: memory access is always dramatically faster than storage access, and SQL Server is engineered to maximize and optimize the use of the buffer pool and other caches to reduce IO. 13.6:1 is very disappointing, and Microsoft should do much better to compete on this.

The Benefits of a Well Documented API That Works Well with Python

I wanted to love using PowerShell for Managed Instance. I’m a huge believer in automation and in scripting out tasks for repeatability and reducing manual toil. I did OK with the API for Managed Instance, but just OK. There were always weird things that weren’t documented correctly or which were hard to get to work. Using LLMS to generate code had similar problems that I did due to gaps in the documentation. It wasn’t terrible, but it was often frustrating.

Using python to automate tasks with the RDS API has been a lot easier. I started out with a very basic knowledge of python – similar to my PowerShell knowledge. But python has some big advantages: it’s quite readable, and LLMs are optimized to do well with python. The RDS API’s documentation is also pretty darn clear and LLMs seem to be well trained on AWS APIs.

Like any generated code: review, review, review, and test. Then review some more. You have to be extremely careful. But the readability of python and the fact that LLMs are well trained in this area are a powerful combo to work with, and it’s quite fun to script tasks for the RDS API.

Y’all, working with python with AWS’ APIs is like having a superpower.

Fast and Effective Customer Support That Isn’t Painful to Use

I wrote a blog post last year titled How to survive opening a Microsoft support ticket for SQL Server or Azure SQL. It truly sucked to be an Azure customer when I had a problem.

The experience of being an Azure SQL Managed Instance customer is that you constantly feel like the support system is trying to deflect you and get rid of you. You feel that the machine is trying to give you any possible answer, whether or not it’s correct, to make you go away as quickly as possible. In order to get quality help, you’ve got to battle various levels of bad advice until you finally get to someone who has time to figure out what’s going on. The system feels optimized primarily to deflect and minimize costs.

The experience of being a SQL Server RDS customer is that you feel like the support system is optimized to help unblock you as soon as possible so that you can successfully use more RDS SQL Server. You have both a client team who can help you and then folks you work with through the ticketing system, and these folks can work together. There is some paperwork occasionally and some waiting sometimes, but by comparison it works pretty darn effectively most of the time. I don’t dread opening a support case at all.

Smart people work in both of these systems. And both of these systems are optimized around profit: Azure feels like it’s trying to reduce costs, AWS feels like it wants to help you spend more. That second experience is a lot more effective and pleasant for customers.

The problems with Azure support aren’t about the employees who provide the support: it’s that the system very much feels to the customer like it’s a cost center that is trying to be squeezed down to the cheapest penny. I don’t think this serves Microsoft well, and it certainly doesn’t help them sell more managed database services to existing customers.

Microsoft, Please Start to Compete on These

As a database person, I want cloud options for managed databases to be more competitive with each other on the basics: performance, automation, and customer service. Right now, for managed SQL Servers in the cloud, AWS RDS has a much better offering for performance and management because of the reasons above.

But these are all very fixable things. Microsoft can make their offerings much stronger in this area, and I hope that they make the investments needed to do so.

Why Won't PostgreSQL Use My Covering Index?

help@KendraLittle.com (Kendra Little) — Thu, 26 Jun 2025 19:43:14 -0400

Dear Postgres, Why won’t you use my covering index?

Lately I’ve been learning to tune queries running against PostgreSQL, and it’s pretty delightful. One fun question that I worked through struck me something that most Postgres users probably encounter one way or another: sometimes you may create the perfect index that covers a given query, but the query planner will choose to ignore it and scan the base table.

Why in the world would it do that?

The Perfect Index

Let’s say you have a relatively large table – something like the Posts table in the Stack Overflow database. You want to run a query to count the number of posts that are questions created after a specific date that have an AnswerCount of 0.

The query for this would use multiple columns:

PostTypeId with an equality predicate (Questions have PostTypeId=1)
AnswerCount with an equality predicate (AnswerCount = 0)
CreationDate with a range predicate (> Date Value)

To “cover” this query, you create a btree index on the table with the index keys PostTypeId, AnswerCount, and CreationDate, in that order.

This index is built and is a nice, beautiful structure with all of those columns. This structure is ordered so that all the rows where PostTypeId = 1 are together, then they are ordered by AnswerCount, so we can quickly find everything where AnswerCount = 0 for our questions. The third ordering column is CreationDate, so we can seek and find all the rows greater than the value we are looking for and count them.

This index seems to cover everything we need in the query.

But It Might Not Get Used

I had a situation similar to this: I created the ideal index for a simple query. Then I asked Postgres for an explain plan– basically to run the query and show me how it executed it with some details.

It did so, and showed me that it didn’t use my index: instead, it scanned the base table with a sequential scan.

I re-checked my index. IT WAS PERFECT. I ran the explain plan again – maybe I was mistaken? Nope, sequential scan against the heap.

What If I Force It to Use It?

PostgreSQL has some costing settings that influence whether the query planner chooses to use nonclustered indexes:

seq_page_cost: “Sets the planner’s estimate of the cost of a disk page fetch that is part of a series of sequential fetches. The default is 1.0.”
random_page_cost: “Sets the planner’s estimate of the cost of a non-sequentially-fetched disk page. The default is 4.0. …Reducing this value relative to seq_page_cost will cause the system to prefer index scans; raising it will make index scans look relatively more expensive.”

Now, my index was perfect in my mind, but I went ahead and ran SET random_page_cost = 1.1; to change this for my session.

I then re-ran explain analyze for my query and I saw that PostgreSQL used my index.

But I also saw that the execution time for my query was much slower than it had taken to generate the explain analyze plan that used the sequential scan of the table.

I’d forced the query planner to use my index, but it made the query slower.

Where did This Random IO Come From?

Thinking a little about the setting I’d tweaked, I realized that I’d only gotten the query planner to use the index by saying that random IO was not such a big deal. It clearly had rejected the index because it didn’t want to do random IO.

So why wasn’t the index “covering” my query? What would it need outside of the index?

This is covered nicely in the docs on Index-Only Scans and Covering Indexes:

But there is an additional requirement for any table scan in PostgreSQL: it must verify that each retrieved row be “visible” to the query’s MVCC snapshot, as discussed in Chapter 13. Visibility information is not stored in index entries, only in heap entries…

PostgreSQL tracks, for each page in a table’s heap, whether all rows stored in that page are old enough to be visible to all current and future transactions. This information is stored in a bit in the table’s visibility map. An index-only scan, after finding a candidate index entry, checks the visibility map bit for the corresponding heap page. If it’s set, the row is known visible and so the data can be returned with no further work. If it’s not set, the heap entry must be visited to find out whether it’s visible, so no performance advantage is gained over a standard index scan.

I was querying a table where a significant amount of data has been changed, and I was running a query that was going to look at a lot of rows.

When I had a higher value for random_page_cost, the query planner realized that it could use the nonclustered index to seek to the data, but that it was going to have to check a whole lot of of corresponding heap pages, which was a lot of random IO.

Tidying Up My Tuples

Vacuuming is the process in Postgres that helps with this. Vacuumming Basics explains that the vacuum process is essential for multiple purposes:

To recover or reuse disk space occupied by updated or deleted rows.

To update data statistics used by the PostgreSQL query planner.

To update the visibility map, which speeds up index-only scans.

To protect against loss of very old data due to transaction ID wraparound or multixact ID wraparound.

Item 3 – that’s our issue.

Sure enough, I looked at the tables used by my query and realized that although the autovacuum process was enabled, the default thresholds were high enough that a whole lot of data had changed in this table without the process kicking in.

I ran a manual vacuum against the table and found that executing my query was now much faster when it used the nonclustered index, and that the query planner was willing to use the nonclustered index when my session had higher values for random_page_cost.

I’ll dig in more in future posts on considerations I’ve found with tweaking these settings, as well as autovacuum and autoanalyze settings.

Fixing OPTIMIZATION REPLAY FAILED in Query Store: When Plan Forcing Breaks in SQL Server 2022+

help@KendraLittle.com (Kendra Little) — Wed, 30 Apr 2025 11:04:28 -0400

Forcing plans with Query Store can be a powerful tool— until it mysteriously fails. In real production systems, plan forcing sometimes just… doesn’t work. One common culprit is the cryptic OPTIMIZATION_REPLAY_FAILED error.

If you’re hitting OPTIMIZATION_REPLAY_FAILED, try re-forcing the plan using @disable_optimized_plan_forcing=1.

The Syntax You May Be Looking For

How to force a query plan without optimization replay:

EXECUTE sys.sp_query_store_force_plan
 @query_id = xxx,
 @plan_id = yyy,
 @disable_optimized_plan_forcing = 1;

Won’t it be great if Query Store on secondaries ever becomes a supported thing and we can use the @force_plan_scope parameter?

Quick Recap of Failed Forced Plans

Sometimes when you force a query plan, SQL Server has a problem enforcing the forcing. If plan forcing fails, the query still executes successfully:

“When a plan is forced for a particular query, every time SQL Server encounters the query, it tries to force the plan in the Query Optimizer. If plan forcing fails, an Extended Event is fired and the Query Optimizer is instructed to optimize in the normal way.” (docs for sp_query_store_force_plan)

When query plan forcing fails, the sys.query_store_plan view will contain some value for last_force_failure_reason_desc.

There is no benefit from leaving a query in a state where plan forcing is failing, so I’ve shared a script that finds and un-forces failed plans.

I’ve implemented versions of this that log the queries/plans found in a failed forced state along with the reason why plan forcing failed.

Many of those queries have OPTIMIZATION_REPLAY_FAILED as the reason.

Optimization Replay is Automatically on in SQL Server 2022+, Even If You’re Not at Database Compatibility Level 160

In my opinion, that error message should say OPTIMIZED_PLAN_FORCING_FAILED, because “optimization replay” is part of the feature named Optimized plan forcing:

Optimized plan forcing reduces compilation overhead for repeating forced queries… Once the query execution plan is generated, specific compilation steps are stored for reuse as an optimization replay script.

Optimized plan forcing (aka optimization replay) occurs in SQL Server 2022 and higher. It is enabled by default and works even if you are using a lower database compatibility level – for example, I regularly see this on SQL Server 2022 with db compat level 150.

Three Ways to Disable Optimized Replay

You can disable optimized plan forcing/ optimized replay in three different ways:

At the query level, using the DISABLE_OPTIMIZED_PLAN_FORCING query hint
At the database level, using a database scoped configuration (aka disable with ALTER DATABASE SCOPED CONFIGURATION SET OPTIMIZED_PLAN_FORCING = OFF)
When forcing a query plan by using the sys.sp_query_store_force_plan stored procedure with @disable_optimized_plan_forcing=1

I wish that when one forced a plan and there was an issue with the replay script, SQL Server would simply fall back to not using a replay script for plan forcing for that query. That doesn’t happen – you need to TRY to force a plan, then detect if plan forcing fails and for what reason. If you find it’s an optimization replay issue, you need to try to re-force the plan with a different parameter.

Or, if you don’t have time for all that, you can disable optimized plan forcing at the database level.

What If It’s GENERAL_FAILURE?

If query plan forcing is failing with the message GENERAL_FAILURE, check out my post on that topic.

TL;DR

If plan forcing fails with OPTIMIZATION_REPLAY_FAILED, re-force the plan using @disable_optimized_plan_forcing=1. It’s common in SQL Server 2022+, even at lower compatibility levels. Always check last_force_failure_reason_desc.

Comparing Single Column, Multi-Column, and Filtered Statistics in SQL Server

help@KendraLittle.com (Kendra Little) — Sun, 13 Apr 2025 14:04:35 -0400

Statistics in SQL Server are simple in theory: they help the optimizer estimate how many rows a query might return.

In practice? Things get weird fast. Especially when you start filtering on multiple columns, or wondering why the optimizer thinks millions of rows are coming back when you know it’s more like a few hundred thousand.

In this post, I’ll walk through examples using single-column, multi-column, and filtered statistics—and show where estimates go off the rails, when they get back on track, and why that doesn’t always mean you need to update everything with FULLSCAN.

I’m Using the Stackoverflow2013 Sample Database at Compat Level 160

To get started, I’ve dropped all my existing indexes and statistics. I’ve created a helper object that lists all the statistics on a given table.

ALTER DATABASE StackOverflow2013 SET COMPATIBILITY_LEVEL = 160;
GO

USE StackOverflow2013;
GO

CREATE OR ALTER PROCEDURE
 dbo.ListStatistics
(
 @ObjectName sysname
)
AS
BEGIN
 SET NOCOUNT ON;

 SELECT
 statistic_name = s.name,
 leading_column = c.name,
 auto_created = s.auto_created,
 user_created = s.user_created,
 has_filter = s.has_filter,
 filter_definition = s.filter_definition,
 no_recompute = s.no_recompute
 FROM sys.stats AS s
 JOIN sys.stats_columns AS sc
 ON s.object_id = sc.object_id
 AND s.stats_id = sc.stats_id
 JOIN sys.columns AS c
 ON sc.object_id = c.object_id
 AND sc.column_id = c.column_id
 WHERE sc.stats_column_id = 1
 AND s.object_id = OBJECT_ID(@ObjectName)
 ORDER BY
 s.name;
END;
GO

Before I get started, the dbo.Posts table has a single statistic.

This statistic is associated with the clustered index on the Id column. Whenever an index is created in SQL Server, a statistic is created along with that index.

Run Our Example Query, Which Will Auto-create Single Column Statistics to Help with Estimates

Our example query has three predicates, on PostTypeId, ParentId, and AnswerCount.

To simplify things so auto-parameterization and query plan reuse don’t involved, I’ve added an OPTION (RECOMPILE) hint to the query.

SELECT
 Id
FROM dbo.Posts AS p
WHERE p.PostTypeId = 1
AND p.ParentId = 0
AND p.AnswerCount = 0
OPTION (RECOMPILE);

When I give this a run, the query plan shows…

SQL Server estimates that 3,200,270 rows will be returned.
305,761 rows are actually returned– only 9.5% of what was estimated. That estimate was very high.

I re-run dbo.ListStatistics and see that three column statistics were automatically created, one on each of the three predicates in my query. I have the AUTO_CREATE_STATISTICS setting enabled for the StackOverflow2013 database– that’s a default setting, and it allows SQL Server to create these small helper objects to put together this estimate.

Each of these statistics took a sampling of rows from the table and used that sampling to create a histogram with “steps” describing what values are in the table and how many rows exist for some values. This is stored along with metadata: how many rows were in the table at the time of the sampling? How many rows were sampled? About how many unique values exist in the column?

For example, here is a view of the histogram for the automatically created single column statistic on the PostTypeId column:

Based on a sampling of the data, there were estimated to be 5,977,672 rows in dbo.Posts with PostTypeId=1 at the time the statistic was auto-created.

The column statistics created for ParentId and AnswerCount similarly describe how data is distributed in those columns.

SQL Server can use these statistics to make estimates, but note that none of these statistics describe data distributions for all three columns when they are used together.

When We Create an Index with Multiple Key Columns, It Creates a Multi-column Statistic to Go with It

Let’s say we have an index on the first two columns in our predicate. Here’s the index:

CREATE INDEX
 ix_Posts_PostTypeId_ParentId
ON dbo.Posts
(
 PostTypeId,
 ParentId
)
WITH
 (DATA_COMPRESSION = PAGE);
GO

I’ve created a nonclustered index that does not fully “cover” my query: it has two out of three predicates.

I run my query again, and SQL Server does not choose to use my new index. Its row estimate is slightly different:

Before, SQL Server had estimated 3,200,270 rows would be returned.
Now, SQL Server estimates 3,212,350 rows will be returned (12,080 more)
This estimate is still very high, only 305,761 rows are returned (the same, rows in the database are not changing)

Why is this estimate just a little bit higher?

If I use DBCC SHOW_STATISTICS to describe the statistic associated with the index, it does describe a little bit about more than the first column, but not much. Important things to notice are:

This statistic was created when SQL Server created the index, and it had to look at every row in the column to do that. It used that information from all the rows (rather than a sampling), so the data in the statistic is a little different.
The statistic has a “Density vector” section, which describes the average selectivity of column values—in other words, how distinct the values tend to be.
- It gives the inverse of the number of distinct (PostTypeId, ParentId) pairs.
- For example, if the all density value is 1.779737E-07, that implies there are about 5.6 million distinct (PostTypeId, ParentId) combinations (1/.0000001779737= 5,618,807).
  - There are 1,7142,169 rows in the table, which implies that any given PostTypeId, ParentId column has about 3.05 rows for it IF there is a very even distribution of values.
The histogram of the statistic is only on the leading column in the index, which in this case is PostTypeId. PostTypeId has a pretty limited set of values.
- The histogram shows 6,000,223 values for PostTypeId = 1. In the column statistic on PostTypeId that was based on sampling, this estimate was 5,977,672. The estimate has gone up by 22,551.

In this case, it looks like the fact that we have a multi-column statistic didn’t really give the optimizer too much new information it thought was useful.

Yes, the density vector implies that there are 3.05 rows on average for any given combination of PostTypeId and ParentId values on average, but the estimate is way higher than 3 rows. It looks like our estimate is still based on histogram values, and the estimate went up a little bit because the histogram for the new index– which is only on the leading column, PostTypeId– estimated a slightly higher number of rows for PostTypeId = 1.

Create a Multi-column Filtered Statistic Tailored to the Query

Now, I could create a multi-column filtered index that is perfect for my query. But let’s say that this query doesn’t run that often, and I don’t want to create an index just for it. However, I want to see if SQL Server might be smarter if I gave it a statistic.

I create a statistic that is very targeted for my query, like this:

CREATE STATISTICS
 FilteredStats_Posts_PostTypeId_ParentId_AnswerCount
ON dbo.Posts
(
 PostTypeId,
 ParentId,
 AnswerCount
)
WHERE PostTypeId = 1
AND ParentId = 0
AND AnswerCount = 0;
GO

One thing I’ve noticed is that when you create a statistic like this, it won’t automatically make queries recompile. I have an OPTION (RECOMPILE) clause in my query, though, so I don’t have to worry about that.

Rerunning my query, I get the same shape plan, but the estimates are much more accurate:

With the filtered multi-column statistic, SQL Server is estimating that 306,267 will be returned, very close to the actual rowcount of 305,761.

This doesn’t make much of a difference in this simple case, but if I was joining to this table and the data was flowing into an operator that needed to allocate memory, like a hash join or a sort, this estimate could make a difference.

Real world talk: I don’t go around creating statistics like this “just in case”, and I wouldn’t assume that this will automatically solve a problem– but sometimes this can help in a specific situation where it doesn’t make sense to create an index and poor row estimates are causing a performance problem.

The Query Plan Depends on Which Version of the Cardinality Estimator (CE) I’m Using

SQL Server has two cardinality estimators: the “legacy” cardinality estimator, which applies to database compatibility level 110 and below (or if you have it enabled as a database scoped configuration), and the “doesn’t really have a name”/newer cardinality estimator.

The legacy cardinality estimator is more likely to assume that data distributions on different columns are independent of one another.

If I have the filtered statistic created and the two column index created, and I run my query using the legacy cardinality estimator, SQL Server decides to use the non-covering index and do a nested loop lookup to pick up the AnswerCount data (which is not in the index).

The row estimate for how much will come out of the nested loop operator is very accurate: 306,267 rows (actual is 305,761).

However, SQL Server estimates that 750,710 rows will come out of the index on PostTypeId, ParentId, whereas it actually had 6,000,223 rows returned from the operator.

Tips and Takeaways About Statistics

Statistics help SQL Server guess row counts.
Statistics are estimates, and close-enough is usually fine. Don’t chase perfection.
Auto-created single-column stats have a histogram that describes how data is distributed in one column at a time. If your query filters on multiple columns, SQL Server has to guess how those filters work together.
Multi-column stats (created with multi-column indexes, or manually) only have a histogram for the leading column.
Filtered statistics can help with tricky queries where the estimate is way off and you don’t want (or need) a new index. If you match the filter exactly, the estimate can get much better.
Don’t assume “bad estimate” means “bad stat” or “bad plan”. Estimates can be off and still lead to a good plan. Make sure to verify if your problem has to do with “statistics were wrong” or if it’s a case of “the statistics were fine for the values the query plan was compiled with, but then the plan ran with different values for parameters” (a parameter sniffing problem).
Avoid the “just fullscan all your statistics” trap. Updating statistics with fullscan all the time burns IO and slows down over time. Becoming dependent upon this can leave you in a reactive position where you regularly have poor performance.
The version of the cardinality estimator matters. Sometimes the legacy cardinality will work better for a query. Sometimes it won’t.

All Eyes on the Wrong Problem: How Mitigations Distract from Real Performance Pain

help@KendraLittle.com (Kendra Little) — Sun, 06 Apr 2025 00:00:00 +0000

🔥 UPDATE: Microsoft has announced the general availability of the Next-gen General Purpose service tier for Azure SQL Managed Instance, which includes improvements to I/O latency, IOPS, and transaction log throughput. This post describes the original General Purpose blob storage. You don't want that.

The biggest lesson I’ve learned from helping folks manage data in Azure is this: if you’ve got a truly terrible problem you’d rather people didn’t notice, a great way to hide it is by educating your support staff and users about something bad but not AS terrible— something with a small mitigation—and constantly refocusing them on that.

The user base— and even your own support staff— will think that anyone who talks about the bigger issue just doesn’t understand how to fix the “known” problem.

This is the story of Azure General Purpose storage for Azure SQL Managed Instance and Azure SQL Database.

Flaw 1: Consistent Lousy Storage Performance

Here’s the ‘really bad but you can mitigate it a little’ thing about General Purpose storage in Azure SQL:

In the General Purpose service tier, every database file gets dedicated IOPS and throughput that depend on the file size. Larger files get more IOPS and throughput… There’s also an instance-level limit on the max log write throughput (see the previous table for values, for example 22 MiB/s), so you might not be able to reach the max file throughout on the log file because you’re hitting the instance throughput limit. Microsoft Docs

There’s a table in the document above that describes the IOPs / throughput levels for files, and why you need to grow them to 130 GB, 514 GB, 1026 GB, or more.

The larger your files, the more you pay, so this is not only inconvenient, but also expensive.

If you contact Microsoft support with any question about storage performance, they will fixate on this weird, complex limitation.

But this is far from the worst storage problem in Azure SQL General Purpose.

Flaw 2: Intermittent Atrocious Storage Performance

There’s a footnote on the resource limits above, which reads:

1 This [Storage IO latency measurement] is an average range. Although the vast majority of IO request durations will fall under the top of the range, outliers which exceed the range are possible.

It’s unclear what they mean by “vast majority”, but let’s be generous and say it’s 95% of requests. Now imagine that 5% of your database storage requests take more than 15 seconds to complete. In fact, I’ve observed this to be between 15 and 60 seconds. 60 seconds. Serially.

Stop and wait for 60 seconds. That’s a real long time.

Now think about what happens when a storage request underneath a database waits that long. The query may be holding locks that block things. All sorts of activity may back up. Things get real, real bad.

Even if it’s much smaller than 5%– in fact I think on average this probably happens to a given database about 10 times a week – you don’t want to run a database on storage like that. Not even an unimportant database. Particularly for what you pay for Azure SQL Database or Managed Instance.

This is beyond alarming, but every time a Microsoft employee reads my post on how Azure SQL Managed Instance Storage is Regularly as Slow as 60 Seconds, they send me an email, direct message, or leave me a comment about how I’ve missed the “bad” issue and that Azure SQL storage performance varies by file sizes. They don’t ask if it’s related, either, they’re confident I’m ignorant of the issue and there’s a way to make this better. This happens so often that I’ve had to update the post to add an explicit disclaimer for Microsoft employees.

In fact, the storage stall problem isn’t mitigated at all by growing data or log files. You can have the largest file allowed by General Purpose storage and you’ll still see IO freezes of 15-60 seconds happen regularly on them over time, just like the smallest files. You will see it when you’re so far under the IOPs and throughput thresholds that it’s laughable. You will see it in a can, you will see it in a ham; You will see it while you troubleshoot with every trace and plan; File size won’t change a thing—it’s not the magic key; You’ll still get hit with storage freezes unpredictably.

But the fact that people know about the “weird file size problem” blocks them from understanding this. Folks regularly argue with me when I let them know that I’ve received confirmation from the Managed Instance product group that increasing file size will not mitigate or stop the storage stalls. They are sure that if you grow the files it will make the issue better.

When the Workaround Becomes Misdirection

The workaround– “grow the files! make ’em huge! pay more for maybe better IOPS!” has become the performance placebo of Azure SQL support. It gives people something to recommend. It creates the illusion of understanding.

But the workaround creates tunnel vision. “THIS is the performance problem. We’ve solved it. Move along.”

Meanwhile, the nastier problem remains massively disruptive to users, and Microsoft support won’t even acknowledge it.

Cognitive Biases

This all lines up with common cognitive biases:

The Streetlight Effect: people keep poking at the file size thing because it’s the only issue illuminated by documentation and support scripts.
Maslow’s Hammer: once the workaround exists, suddenly every performance problem starts to look like it must be file-size related.
Normalcy Bias: hey, 60-second storage stalls sound too extreme to be real, right?

And when you show someone new evidence, Semmelweis Reflex kicks in—we reject new information out of habit, because it contradicts what we’ve been taught.

In a complex world, it’s easy to fall into a feedback loop that feels logical in the moment, but keeps us stuck. Our brains prefer a known explanation to a messier reality.

What to do About It?

First off, if you’re using Azure SQL General Purpose and you’re experiencing weird lockups, long-running queries that don’t make sense, or sessions waiting on I/O for disturbingly long periods—you’re not imagining things. You can try growing your files, but it’s not going to fix the whole issue. And sorry, but Azure Support isn’t going to understand the issue unless you escalate about six levels, and then you’re going to be told to just wait for GPV2.

More generally, we need to keep our minds open to new information.

Ask the awkward questions. Use data to measure performance yourself, and break it down critically. When you find something that doesn’t line up with the documentation or the community folklore, go ahead and speak up.

Our most valuable technical trait isn’t knowing database internals or mastering syntax. It’s being persistently curious enough to keep asking questions when the data doesn’t all line up.

100 Things I Hate About Views: Undeclared Data Types in Columns

help@KendraLittle.com (Kendra Little) — Fri, 14 Mar 2025 17:44:51 -0400

Views let you do dumb things by accident in SQL Server. Then they make you have to think way too hard to fix them.

Most of the time when people create views, they start by refining a SELECT query, then turn it into a view. People also often create multiple views that pull different slices of data and UNION the results together.

Combined, these two things easily lead to undeclared datatypes in views with problematic implicit conversions.

SQL Server Lets You Define a View with Columns with Undeclared Data Types

You can even have it return a null if you’re going to UNION that view with other things.

It doesn’t look very weird at all, it just looks like this…

CREATE OR ALTER VIEW
 dbo.i_am_view
AS
SELECT
 some_column = NULL
 /* column2,
 column3
 FROM dbo.Sometable, etc
 */
GO

Here’s the table and its rows:

CREATE TABLE
 dbo.i_am_table
(
 some_column varchar(100) NOT NULL,
 last_modified_date datetime2(0) NULL
);
GO

INSERT INTO
 dbo.i_am_table
(
 some_column,
 last_modified_date
)
VALUES
 ('1', '2023-01-01'),
 ('ONE', NULL),
 (NULL, '2024-01-01'),
 ('', '2025-01-01');
GO

Select from the View and Union with the Table

Here’s the query

SELECT
 some_column,
 last_modified_date = ''
FROM dbo.i_am_view AS v

UNION

SELECT
 some_column = t.some_column,
 last_modified_date = t.last_modified_date
FROM dbo.i_am_table AS t
WHERE t.last_modified_date IS NOT NULL
ORDER BY
 last_modified_date;
GO

The Results are Odd

Notable bits:

Row 1 came from the view. The last_modified_date column, which was also not given a data type in our query, returned January 1, 1900– it got a magic default value.
Row 4 has 0 as the value for some_column. This was an empty string in the table, but it’s been converted to a numeric 0.

The Query Execution Plan Explains More

Let’s look at that warning on the SELECT operator, plus that compute scalar operator after we read the table.

We’re Warned About an Implicit Conversion

I find these messages really hard to read with the text all crammed together. Here it is with a lot of spacing added:

 Type conversion IN expression (
CONVERT_IMPLICIT
(integer,
[views].[dbo].[i_am_table].[some_column]
,0)
) may affect "CardinalityEstimate" IN query plan choice

SQL Server decided that it needed to convert the some_column values in dbo.i_am_table from the datatype they are stored as– VARCHAR(100) – into the INT datatype.

Looking at the compute_scalar operator confirms this.

Expr1005 is defined as the result of that CONVERT_IMPLICIT function, which is converting values from some_column to integers. Expr1005 is returned from this operator along with the last_modified_date column.

If We Remove the Where Clause, the Query Fails

If we let that row where some_column has the value “one” through, the query fails with the error:

Msg 245, Level 16, State 1, Line 52 Conversion failed when converting the varchar value ‘ONE’ to data type int.

I guess we can’t implicitly convert words to integers.

Why an Integer?

When we created the view, we defined it with:

SELECT
 some_column = NULL

We didn’t give SQL Server anything to go on, so it guessed int.

The same thing happens if we use this syntax to create a temp table.

We shouldn’t leave this to chance

I wish SQL Server wouldn’t let you create a view this way, but it does. However, we’re much better off specifying a data type, which we can do with the syntax:

CREATE OR ALTER VIEW
 dbo.i_am_view
AS
SELECT
 some_column = CAST(NULL AS varchar(100));
GO

This avoids errors with data type conversions, not to mention unexpected results and performance issues with type conversions.

What the Decline of SQL Server Quality Means for Developers and DBAs

help@KendraLittle.com (Kendra Little) — Mon, 10 Mar 2025 08:55:51 -0400

‘Is it just me, or is SQL Server quality slipping?’

I asked myself that question for couple/few years until I faced up to it: SQL Server is well into a period where Microsoft investment is waning, and Microsoft regularly isn’t able to deliver the features they promise.

Feature Announcements ≠ Usable Features

Remember when folks waited for Service Pack 1 for SQL Server releases before trying a feature, because they weren’t sure about the quality?

We don’t have service packs anymore, but these days a lot of features wouldn’t even be available by Service Pack 3, much less Service Pack 1.

We are in an era when Microsoft announces features for SQL Server that they don’t deliver for years– or perhaps at all. For example, Query Store on readable secondaries was a SQL Server 2022 feature that we’ve still never seen in a production-ready state: it remains completely unavailable in Azure SQL offerings. It stays in a “preview” state that “is not intended for production deployments” in the boxed product years after the 2022 release.

Similarly, we saw other flagship features of SQL Server 2022, like the ability to fail over to Azure SQL Managed Instances, take years to crawl out of preview.

Azure SQL Managed Instance: Storage That Freezes for 60 Seconds

Azure SQL Managed Instance itself is one of the sharp indicators of a lack of Microsoft investment.

The General Purpose tier of Azure SQL has an incredibly bad storage latency problem– storage regularly stalls up to 60 seconds regardless of utilization or configuration patterns, and there’s nothing you can do as a user to prevent this.

That’s essentially downtime, and accepting this as “normal” in a hosted database product is, well, bizarre. Yet Microsoft can’t get the newer version of the storage, GPV2, over the line and out of preview after announcing it a year ago. Note to Microsoft employees who think you can fix this by growing file sizes: you can’t. The fact that smaller files have worse IO performance is a completely separate bad thing about the product, users have to deal with both. Even the largest files will regularly see 60 second IO stalls.

Lingering SQL Server Bugs are Piling Up, Long Unfixed

It’s more difficult than ever to get SQL Server bugs diagnosed and fixed.

Take the query hash bug—where SQL Server randomly replaces query_hash values, making performance tuning nearly impossible (source). This bug has been lurking for years and impacts SQL Server 2008 through the latest versions.

Similarly, SQL Server bugs with statistics with metadata errors that cause queries to fail on readable secondaries have been around since SQL Server 2012. There are a host of known problems with In-Memory Metadata for tempdb— which maybe is why it remains completely unavailable for Azure SQL Database and Azure SQL Managed Instance, it’s hard to say. And online index rebuilds aren’t always online.

And these are just the ones that I hit in a regular workday.

All Microsoft Roads Lead to Fabric

Looking in from the outside, it seems clear that Microsoft’s focus has shifted: when it comes to data, the priority is Microsoft Fabric.

Investment in data technologies are increasingly focused on the AI goldrush and quest to beat Databricks.

It also feels like Microsoft has, at an organizational level, decided that SQL Server doesn’t need to evolve much more. When is the last time you heard of a SQL Server feature that felt groundbreaking and innovative to you? From my external viewpoint, it seems like Azure SQL Database Hyperscale tried to take on AWS Aurora, and after a crowd of users didn’t materialize then product innovation sharply dwindled.

Will the Pendulum Swing Back?

Honestly, in this case, I’m not sure.

I’m quite positive that Microsoft will keep releasing new versions of SQL Server and keep selling licenses for quite a long time. The product has a large enough user base and enough functionality that it will keep making money for years, even if it goes onto a full mode of quiet life support with only small features added and bug fixes released.

Will innovation and strong investment actually swing back to SQL Server, though? If this happens, I think it’s going to be a while.

What This Means for Developers and DBAS

So SQL Server is receiving limited investment: what does that mean to DBAs and Developers?

It doesn’t necessarily mean work is less interesting for database folks– I work with SQL Server databases every day, and I love my job. It remains fascinating and full of tough problems.

But Microsoft’s declining level of investment in SQL Server will change things for those of us who use the product:

You may adjust your technical focus, depending on where you are in your career path: For people earlier in their career cycle, this is a strong signal not to specialize only on SQL Server. If relational databases are your interest, work to get experience with multiple database platforms. Postgres is already common and will continue to grow its userbase.

Slowing upgrade trends: Mirosoft is still planning on releasing SQL Server 2025 (without having finished SQL Server 2022, arguably). The will still push users to upgrade and limit the number of versions they support. However, the pattern of not delivering promised features and the slow rate of addressing bugs means that users will be slower to upgrade services to new major versions. This will, in turn, probably mean users are slower to report new bugs, and the cycle will deepend and become even more sluggish.

You’ll need to be a squeakier wheel: The problem isn’t that no one at Microsoft cares about SQL Server—there are still engineers and product managers who genuinely want to improve it. The issue is that at a high leadership level, investment has shifted elsewhere. If nobody complains and we all seem quite happy to just keep paying for upgrades, why would that ever change? Microsoft prioritizes what makes the most noise internally, and that noise comes from signals—support tickets, customer escalations (particularly to sales teams), and public feedback. If we want long-standing issues fixed and critical features delivered, we need to keep making noise and providing strong signals that product investment matters.

AI Will Eliminate DBA Jobs Faster Than You Think

help@KendraLittle.com (Kendra Little) — Sun, 02 Mar 2025 10:18:54 -0500

I listened to ‘Surviving the A.I. Endgame’ this weekend and realized: I’ve become one of the believers that AI advances are very likely to completely change tech and knowledge roles as we know them over the next 10 years. This is going to dramatically shrink the workforce across MANY roles (and many of those impacted will be outside of the tech sector). It isn’t that people won’t be needed anymore, but far fewer people will be needed. Including people with database administrator (DBA) roles like mine.

Database administrators have been hearing that technological improvements will mean DBAs are no longer needed for 20+ years, though. So, as a DBA, why would I start believing this now?

I’m Not an AI Fanboy. I’m Not an AI Opponent.

This post is about what I believe is likely to happen– not what I want to happen. I’m not labeling this prediction as a “good” or “bad” outcome, either.

This is what my experience has led me to believe is coming.

I’ve been Working with an AI Agent

Lately I’ve been writing a fair amount of Python with an AI agent. If you’ve mostly used AI chat experiences, I think the Agent experience is pretty different. Agents can do things like:

Apply changes for you in multiple files, verify that they work, then let the user modify them, accept them, or reject them.
Handle multiple complex steps – like verifying how it is doing at a task, noticing problems in its approach, and adapting its approach
Report on what it is doing, and take further input from the user while it is still working.

In many ways I feel like I can see the software team behind the agent do things as time goes on, because I feel like the agent is gradually getting better at learning when and how to identify its own mistakes before telling me it’s done with a task.

I’ve also noticed that there’s a learning curve for me: as I get better at providing the right context from files and sources for tasks, my results get better. As I get better in understanding how as a user I can shape the Agent’s behavior, I can give the Agent instructions that make it better at handling things it doesn’t start out being very good at.

The agent dramatically increases my ability to get things done, to the point where it’s not too hard for me to see how my role in the situation may rapidly become optional, then unnecessary.

Reflections on Responding to Database Performance Incidents

“Happy databases are all alike; every unhappy database is unhappy in its own way.” Anna Karenina principle

While there are plenty of bizarre bottlenecks and performance situations that one can find with databases, responding to database performance incidents generally involves doing a set of very similar tasks to collect information, diagnose what is going on, and suggest next steps and mitigations. There is a fairly short list of “frequent fliers” that are very commonly causes of bad performance that one can check for as part of this process, and verify them or rule them out.

This set of tasks and diagnostic rules are fairly complex. I’ve had plenty of success in teaching other people these tasks and rules, but up to this point I didn’t think that this could be automated.

After my recent experience working with an Agent I don’t think it’s gonna be long before we get there, though.

An AI Agent Doesn’t have to be Perfect– Just Make Fewer Mistakes Than a Person

DBA jobs won’t be the first to be replaced by AI agents because of the risk associated with data loss. However, I think the quality of AI Agents will become high enough that it starts to make sense.

Agent quality isn’t the only factor, however. Cost is also a factor in the algorithm. AI agents can be expensive, but so are human employees.

But Kendra, AI is Bad at SQL

Oh, I agree, I have plenty of examples of LLMs spewing utter nonsense and writing garbage SQL.

My views are largely informed by my experience using an AI Agent to get DBA projects done that require lots of automation. I’ve certainly generated SQL scripts in the course of this, but they’ve been fairly simple and related to working with metadata. The AI Agent is already pretty excellent at all of this stuff.

My belief is that the tooling will continue to improve rapidly and that this will cross into other areas. I think different languages/work areas will have agents that are tuned for those areas. I think it will get good at even complex SQL. And I think this evolution is going to continue to be quite rapid.

What’s a DBA to Do?

Depends on your personality, really. If you’re a startup-y type, there’s probably decent money to be made with jumping on the AI Agent Tooling Building Bandwagon.

If you’re not a startup-y type, there’s plenty of time to gather more data and see if my little 10 year prediction here is totally bonkers or not.

How to Stop SSDT / Database Projects / SQLPackage from Modifying Database Options

help@KendraLittle.com (Kendra Little) — Thu, 13 Feb 2025 00:00:00 +0000

SQL Server’s free state-based version control tooling was introduced under the ‘Data Dude’ brand, then became known as ‘SQL Server Data Tools’ (SSDT). Its extension for the (now dying) Azure Data Studio IDE is called ‘SQL Database Projects’. If you need to find documentation, you often need to know to search for specific component names like SQLPackage.exe, which is a command line utility used to deploy SSDT Projects AKA SQL Database Projects.

Database Projects Control More Than Objects Inside the Database

By default, deploying a SQL Database Project with SQLPackage.exe doesn’t only deploy the schema of the tables, procedures, and functions in the project. It will also deploy select attributes of SQL Server databases themselves, things like page verification settings, whether snapshot isolation is enabled, and some Query Store settings.

If there’s a complete list of the database properties that are version controlled with SQL Database Projects, I haven’t found it. Update: after publication of this post, a doc appeared.

Sometimes You Don’t Want to Overwrite Database Properties on Your Target Databases

For various reasons, you may not always want your project to deploy database settings everywhere. Perhaps you deploy your project to many single-tenant databases and you are gradually rolling out changes to settings across your environment, for example.

In this case, I’ve found that running SQLPackage.exe with the /p:ScriptDatabaseOptions=False parameter seems to work: when this is set, sqlpackage.exe won’t overwrite database properties that are controlled by the database project but vary from their definition in source.

Avoid ExcludeObjectType(s) for This

But beware: there are some other documented options like /p:ExcludeObjectType:"DatabaseOptions" which do NOT work for this. I haven’t been able to identify if setting that property, or using the similar ExcludeObjectTypes, has any effect at all, or why those exist. I tested those for a good hour, trying to get them to work and despairing until I found that /p:ScriptDatabaseOptions=False does the trick.

Buyer Beware: Azure SQL Managed Instance Storage Is Regularly as Slow as 60 Seconds

help@KendraLittle.com (Kendra Little) — Wed, 18 Dec 2024 09:26:20 -0500

What are your stories of unbelievably bad performance from cloud vendors? I’ll go first. For years, Azure SQL Managed Instance’s General Purpose Tier has documented ‘approximate’ storage latency as being “5-10 ms.” This week they added a footnote: “This is an average range. Although the vast majority of IO request durations will fall under the top of the range, outliers which exceed the range are possible.”

How approximate is that 5-10 milliseconds, you might wonder? If you use Azure SQL Managed Instance these days, you will regularly find messages in your SQL Server Error log indicating that all data and log files have experienced latency of up to 60 seconds. At least, 60 seconds is the maximum I’ve observed personally, looking in the logs of several customers’ Managed Instances. Could it be worse? Microsoft hasn’t documented a ceiling. My testing shows that this latency occurs randomly to your workload and is not related to your resource usage: using less IO will not make the errors less likely. You have no way to avoid these storage failures (I don’t see how 15-60 second latency is not a failure), and they can occur anytime.

I’m not a mathematician here, but one minute appears to be 6000x higher than that advertised 10ms latency. Is that what a prospective customer would assume when reading the documentation? And when it comes to a hosted database service, is that an acceptable product feature?

A Note for All the Microsoft Employees Who Think This is About File Sizes

I keep getting comments in various places from Microsoft employees who think I’m missing something obvious, so let me say up front here: this issue is NOT mitigated by growing database file sizes.

Yes, the storage in Microsoft’s offerings in Azure is so bad that you have to grow file sizes to different levels to make your general IO performance slightly less terrible.

However, the issue I’m talking about here, where storage will stop responding for your data and log files for up to 60 seconds, will happen regardless of the size of those files. You can grow them all you want, you’ll still regularly see stalls of 45-60 seconds. It is that bad.

What are Reasonable Latencies for Database Storage?

Back in 2013, Paul Randal, who was once a PM on the Storage Engine team at Microsoft, wrote:

Everyone has their idea of what constitutes good or bad I/O latency, and here’s my take:

Excellent: < 1ms

Very good: < 5ms

Good: 5 – 10ms

Poor: 10 – 20ms

Bad: 20 – 100ms

Shockingly bad: 100 – 500ms

WOW!: > 500ms

Paul is talking about average latencies. So let’s think about what average latency ends up looking like in a 5 minute window when you have a 45 or 60 second IO stall on all of the read/write requests to your data and log files. I don’t want to put words in Paul’s mouth, but I think we’re in “what the hell is wrong with this?” territory by his scale.

General Purpose - More Like No Purpose

Microsoft SQL Server, the product, agrees that multiple seconds of stalled IO are an alarming problem and should be considered a serious error for the storage subsystem. Back in 2010, Microsoft’s Bob Dorr shared his excellent SQL Server I/O Presentation. In it, he wrote:

Stalled/Stuck I/O: SQL Server 2000 SP4 added a warning that the I/O was taking too long and appears to be stuck or stalled. When an I/O request is posted (async) the time is kept (sys.dm_io_pending_io_requests) with the tracking information. Lazy writer checks these lists periodically and if any I/O is still pending at the operating system level (FALSE == HasOverlappedIoCompleted) and 15 seconds has elapsed the warning is recorded. Each file will report the number of stalls at most every 5 minutes to avoid flooding the log.

Since a normal I/O request should respond in ~15ms or less 15 seconds is way too long. For example if the I/O request is stalled for 30 seconds and the query timeout is 30 seconds it can cause query timeouts. If the stalled I/O request if for the log it can cause unwanted blocking situations.

If you are seeing these warnings you need to double check the I/O sub-system and use SQLIOSIM.exe to help narrow the problem. It can be anything from the configured HBA queue depth, multi-path failover detection mechanism, virus scanners or other filter drivers.

But these days, dear reader, it could simply be that you are using Microsoft’s Azure SQL offering. You will not be able to take action on any of those things.

Bob Dorr’s warnings are worth reiterating here: stalled/stuck I/O will regularly cause blocking in SQL Server. The longer the duration of the stall, the more blocking can pile up, and this can push the impacts of that storage latency on your workload to far longer than 60 seconds.

How Much does General Purpose Cost Again?

Here are some sample prices for pay-as-you-go in US-East today:

24 vCore, Premium Series Memory Optimized - 16 TB storage - $7,547.47/month - $90,569.64/year
64 vCore, Premium Series Memory Optimized - 16 TB storage - $16,412.74/month - $196,952.88/year

You can spend a lot on something where the storage is regularly slower than a fax machine.

It’s Simple to Find These Errors. Here’s How.

To find these errors, simply open the SQL Server log and search for “longer than 15 seconds.”

You’ll find messages that look like this:

SQL Server has encountered [#] occurrence(s) of I/O requests taking longer than 15 seconds to complete on file [blah blah blah] in database [blah].

The end of the message will report the current highest latency IO, which I’ve seen be as high as 60 seconds on Azure SQL Managed Instance. I’m curious if you ever see it go higher– please let me know in the comments.

Some tips:

It’s incredibly slow to open the logs in Azure SQL Managed Instance, so I recommend using Erik Darling’s free sp_loghunter utility to find these programmatically. This also allows you to copy /paste the message info out.
Azure SQL Managed Instance General Purpose deletes log files after it restarts (ugh), so you may not see the messages shortly after maintenance has occurred. This doesn’t mean the errors haven’t occurred recently, it just means the evidence has been lost. My experience has been that these messages regularly occur, both on read/write instances and failover partners, in different regions.
As mentioned in Bob Dorr’s quote above, you will only see these errors logged every 5 minutes. They will occur for longer than that! You are only seeing a glimpse of the full scope of this problem.

What About Gpv2? Why is General Purpose Still Gpv1? Here’s the Gossip.

These errors occur on the only fully supported version of Managed Instance General Purpose, which Microsoft is now calling GPV1. A new version, GPV2, is in public preview. Microsoft may suggest you use this, but be sure to ask about the fine print. Ask them if there is an outage on GPV2, will Microsoft provide credits for that outage?

This is why everyone is still using GPV1, because it’s fully supported. Preview products are not.

But GPV2 has been in preview for a long time. With these massive problems of latency in GPV1, why hasn’t Microsoft simply made GPV2 fully supported?

I have no secret or non-secret info here: Microsoft hasn’t told me this. And let’s face it, due to a long stream of my complaints and failed attempts to get Microsoft to either fix this storage problem or be transparent about it with their customers, I don’t think Microsoft will be telling me any secrets in the future.

But I can tell you what rumors are going around in the community. These may or may not be true, but they do reflect the reality of what non-Microsoft DBAs and consultants in this space think is the case about why GPV2 hasn’t shipped:

It will destroy the Business Critical offering. For a good while, rumors were that GPV2 was moving so slowly because having better storage in General Purpose would make the more expensive Business Critical offering even less attractive. Personally, I’ve found that Business Critical Managed Instance already isn’t worth the price tag.
Internal wheelings and dealings have the Managed Instance product locked in to pay for General Purpose V1 storage for a long time, and they don’t want to take the financial loss of moving off it soon.

Basically, the community of folks who work with these products outside Microsoft think that GPV2 would work well, but internal politics and pricing and packaging issues keep customers stuck with a product with regular issues with abysmally bad storage performance, with the only alterative being to pay through the nose for Business Critical.

These rumors may or may not be accurate, but they reflect the depression of the profressional community around the Azure SQL offering.

Does Azure SQL Database Also have Regular Storage Latency of Up to a Minute?

As far as I’m aware, Azure SQL Database General Purpose uses the same storage model and will have the exact same issues. Probably the Fabric version is the same.

Unfortunately for users, Azure SQL Database does not provide access to the SQL Server Error Log and does not provide ways to access all the errors in the log. It only provides very limited access into security errors.

This probably means that latencies of up to 60 seconds can also occur on General Purpose Azure SQL Databases, but you’ll have a much harder time proving it.

Unreasonable Customers

I’ve professionally been a champion of Microsoft SQL Server for nearly 20 years. I’ve been proud of the product as it’s grown and worked to help developers and DBAs around the world get the most out of it. I’ve thought for a long time that it’s worth the money.

I don’t believe that regular storage stalls of 15-60 seconds are acceptable for a hosted database product, though. And I also feel strongly that cloud vendors should be up front with customers about expected latencies with their products if they aren’t going to fix them.

I’ve been trying to be a good partner and let Microsoft document this issue before I share my technical observations here publicly. Yesterday, a friend asked if I’d seen the documentation pull request to update the resource limits page. The PR includes the comment:

Added clarification to the IO latency duration range to make clearer that IO requests exceeding the range are possible. (Although not as clear as I’d like - using “compromise language” based on comment from Microsoft Employee–name redacted intended to balance clarity with keeping us out of trouble with unreasonable customers).

I’m not linking to the PR because I don’t think it matters who wrote this. I also don’t particularly care if I’m the “unreasonable customer” here – if thinking regular storage latency between 15 and 60 seconds is ludicrous and should not be kept secret is unreasonable, yeah, I’m REALLY unreasonable. You betcha.

What I think is notable here is the blatant butt-covering. The focus isn’t on what’s best for the customer. The focus is on protecting the corporation FROM customer expectations. We might demand that products work as they are described.

Y’all, if you’re a customer, check your error logs. Be willing to be called unreasonable for asking for something to live up to SQL Server 2000’s standards. Because customers are going to have to force Microsoft to make their hosted versions of SQL Server lived up to the product’s own expectations from SQL Server 2000. Complain to your Microsoft sales reps and cloud architects, and let them know this matters.

Sometimes Users Need to Complain Before Changes are Made

This isn’t the first time it’s taken someone being brutally honest to change something at Microsoft – it was only after Brent Ozar wrote SQL Server 2014 Standard Edition Sucks, and It’s All Your Fault that Microsoft raised the ridiculously low memory limit on Standard Edition from 64GB to 128GB. And I recall that some Microsoft employees were super mad that he dared to write that post at the time.

Please, join me in complaining– perhaps Microsoft can work through whatever internal political blockades exist and push GPV2 into full support. Maybe we can make Azure SQL less of a cautionary tale. From my experience in working in software companies in general, I absolutely believe that customer complaints can be a real asset to the folks inside the corporation who do care about customer experience and making things better. These complaints help them justify getting the resources to make changes. That is something that everything not labeled “Fabric” badly needs.

What are Your Cloud Stories? How Bad does It Get?

Like I said at the beginning: I want to hear your stories. Azure, AWS, Google Cloud, whatever the Oracle Cloud is– do you have a story of cloud performance that’s shockingly slow? I know I’m not alone here, I’d love to hear it.

Query Hash Values Are Meaningless in SQL Server: They May Be Reset to Be the Same Value as the Query Plan Hash

help@KendraLittle.com (Kendra Little) — Sun, 24 Nov 2024 00:00:00 +0000

This is the worst bug I’ve found in SQL Server to date. Previously, my top find was SQL Server Online Index Rebuild sometimes happens offline without warning. This one has taken top slot because it makes my life more difficult on a daily basis.

Background: SQL Server generates a query_hash for each query. This is stored in sys.query_store_query and it’s one of the primary ways you can identify what a query is across different Query Stores, or even the same Query Store over time, as surrogate query_id values get reset if Query Store is cleared or data ages on. The query_hash is a ‘Zobrist hash over the shape of the individual query, based on the bound (input) logical query tree. Query hints aren’t included as part of the hash.’ (Source)

Except that’s wrong. These days it’s NOT always a hash of the query. If you force query plans, or if you use something like Automatic Plan Correction which forces plans, frequently the query_hash value will be bizarrely overwritten by a query_plan_hash value. So sometimes the query_hash ends up being the ‘MD5 hash of the individual plan." (Source) This occurs first in the query_plan XML, then it weirdly invades the Dynamic Management Views in Query Store for all query_hash values for that query_id.

This presents a big challenge identifying queries in Query Store, or writing reliable diagnotic tooling and automation. I’ve reported this, but Microsoft hasn’t acknowledged that it’s a bug yet. At this point they are still asking me if there’s a pattern in how my TSQL is written that causes the query_hash value to be the same as the query_plan_hash.

Dear reader, if my TSQL was that powerful I would be doing MUCH wilder things than stunts with query_hash values.

This post shows a repro of the problem and just how unreliable the query_hash has become in SQL Server 2022 and Azure SQL Managed Instance. It may happen in additional versions, I haven’t tested more broadly than this.

Set Up Our Environment to Reproduce This

I’m using a copy of the StackOverflow2013 database to reproduce this. I’m currently running Developer Edition with @@version Microsoft SQL Server 2022 (RTM-CU15-GDR) (KB5046862) - 16.0.4155.4 (X64).

To get started, I reset basic configuration, make sure Query Store is enabled, clear Query Store, set database compat level to 160, and make sure Automatic Plan Correction is off.

Warning: Automatic Plan Correction automatically forces query plans and it causes this bug to spread rapidly. Especially avoid on relying on query_hash to identify queries if you have that feature enabled. However, the bug occurs with plan forcing even if you’re manually forcing plans, so I’m reproducing that in this post.

USE master;
GO

EXECUTE sp_configure 'show advanced options', 1;
GO

RECONFIGURE;
GO

EXECUTE sp_configure 'MAX degree of parallelism', 8;
GO

RECONFIGURE;
GO

EXECUTE sp_configure 'cost threshold for parallelism', 50;
GO

RECONFIGURE;
GO

ALTER DATABASE StackOverflow2013 SET QUERY_STORE = ON;
GO

ALTER DATABASE StackOverflow2013 SET QUERY_STORE
(
 OPERATION_MODE = READ_WRITE,
 MAX_STORAGE_SIZE_MB = 4000,
 CLEANUP_POLICY = (STALE_QUERY_THRESHOLD_DAYS = 10)
);
GO

ALTER DATABASE StackOverflow2013 SET COMPATIBILITY_LEVEL = 160;
GO

USE StackOverflow2013;
GO

ALTER DATABASE CURRENT SET QUERY_STORE CLEAR;
GO

ALTER DATABASE CURRENT
SET AUTOMATIC_TUNING
(
 FORCE_LAST_GOOD_PLAN = OFF
);
GO

Create Indexes and Procedure for a Parameter Sniffing Situation

Next, I create two indexes and a stored procedure. This is not meant to be beautiful code, it’s code I wrote for a demo on problems that can arise because Automatic Plan Correction doesn’t exclude temp tables, but it works to reproduce this issue as well.

EXECUTE dbo.DropIndexes;
GO

CREATE NONCLUSTERED INDEX
 ix_Posts_OwnerUserId_INCLUDES
ON dbo.Posts
(
 OwnerUserId
)
INCLUDE
(
 Id
);
GO

CREATE NONCLUSTERED INDEX
 ix_Users_DisplayName_INCLUDES
ON dbo.Users
(
 DisplayName
)
INCLUDE
(
 Id
);
GO

CREATE OR ALTER PROCEDURE
 dbo.PostsForDisplayname
(
 @DisplayName nvarchar(40)
)
AS
BEGIN
 SET XACT_ABORT, NOCOUNT ON;

 SET @DisplayName = @DisplayName + N'%';

 SELECT
 user_id = u.Id
 INTO
 #users
 FROM dbo.Users AS u
 WHERE u.DisplayName LIKE @DisplayName;

 /* This query selects FROM the #users temp TABLE
 That means it can USE statistics ON the temp TABLE to recompile IF the TABLE
 has a lot more data IN it than IN the cached plan */
 SELECT TOP (10)
 p.*
 INTO
 #bin
 FROM dbo.Posts AS p
 JOIN #users AS u
 ON p.OwnerUserId = u.user_id
 ORDER BY
 p.Score DESC;
END;
GO

Run the Procedure and Get Two Plans in Cache for the Second Query

We run the following SQL:

EXECUTE dbo.PostsForDisplayname N'Clothilde';
GO

EXECUTE dbo.PostsForDisplayname N'Jon';
GO

There are no posts by users named ‘Clothilde%’, but there are a LOT of posts by users named ‘Jon%’, so a recompile naturally occurs due to statistics on the temporary table.

Run the following query to review the hashes from both the Query Store DMVs and extract the same values from the query plan XML:

SELECT
 query_id = q.query_id,
 plan_id = p.plan_id,
 query_sql_text = qt.query_sql_text,
 query_plan = CAST(p.query_plan AS xml),
 query_hash_from_sys_query_store_query = q.query_hash,
 query_plan_hash_from_sys_query_store_plan = p.query_plan_hash,
 query_hash_extracted_from_plan_xml = CAST(p.query_plan AS xml).value
 (
 N'DECLARE namespace sp="http://schemas.microsoft.com/sqlserver/2004/07/showplan"; (/sp:ShowPlanXML/sp:BatchSequence/sp:Batch/sp:Statements/sp:StmtSimple/@QueryHash)[1]',
 N'varchar(64)'
 ),
 query_plan_hash_extracted_from_plan_xml = CAST(p.query_plan AS xml).value
 (
 N'DECLARE namespace sp="http://schemas.microsoft.com/sqlserver/2004/07/showplan"; (/sp:ShowPlanXML/sp:BatchSequence/sp:Batch/sp:Statements/sp:StmtSimple/@QueryPlanHash)[1]',
 N'varchar(64)'
 ),
 is_use_plan_extracted_from_plan_xml = CAST(p.query_plan AS xml).value
 (
 N'DECLARE namespace sp="http://schemas.microsoft.com/sqlserver/2004/07/showplan"; (/sp:ShowPlanXML/sp:BatchSequence/sp:Batch/sp:Statements/sp:StmtSimple/sp:QueryPlan/@UsePlan)[1]',
 N'integer'
 )
FROM sys.query_store_query AS q
JOIN sys.query_store_query_text AS qt
 ON q.query_text_id = qt.query_text_id
JOIN sys.query_store_plan AS p
 ON q.query_id = p.query_id
WHERE q.object_id = OBJECT_ID(N'dbo.PostsForDisplayname', N'P');
GO

We care about the second query in the procedure here, which has been assigned query_id 7. Query_id 7 has two different query plans, which have plan_ids 9 and 10. Here are the hash values we currently have (extracted from the image):

Query Id	Plan Id	Query Hash sys.query_store_query	Query Plan Hash sys.query_store_plan	Query Hash in Plan XML	Query Plan Hash in Plan XML
7	9	0xD55EEC4C7A81E644	0x44A186BFC52AA053	0xD55EEC4C7A81E644	0x44A186BFC52AA053
7	10	0xD55EEC4C7A81E644	0x56049465CF9FFB95	0xD55EEC4C7A81E644	0x56049465CF9FFB95

This all looks normal and good:

Query_id 7 has a query_hash that is the same in sys.query_store_query as it is in the query plans.
The query_plan_hash is a different value because it’s a hash of the query execution plan, not the query.
The query_hash values in sys.query_store_plan match the values in the query plan XML.

Force a Query Plan, Then Rerun the Stored Procedure for a New Value

Next, we’re going to force the “small” query plan for this– the one we ran for Clothilde which had very few rows in the temp table.

After forcing that plan, we run the procedure for the value J, which is going to use that forced plan but will find a whole lotta rows. Run that twice:

EXECUTE sys.sp_query_store_force_plan
 @query_id = 7,
 @plan_id = 9;
GO

EXECUTE dbo.PostsForDisplayname N'J';
GO 2

Something’s Gone Weird in the Query_hash Value in Our Execution Plan

Rerun the query above retrieving the query_hash and query_plan_hash values from the DMVs and the query plans.

We have a new query plan, and something is starting to look weird:

Here are the values for query_id 7 extracted from the image:

Query Id	Plan Id	Query Hash sys.query_store_query	Query Plan Hash sys.query_store_plan	Query Hash in Plan XML	Query Plan Hash in Plan XML	Is Use Plan Hint in Plan?
7	9	0xD55EEC4C7A81E644	0x44A186BFC52AA053	0xD55EEC4C7A81E644	0x44A186BFC52AA053
7	10	0xD55EEC4C7A81E644	0x56049465CF9FFB95	0xD55EEC4C7A81E644	0x56049465CF9FFB95
7	11	0xD55EEC4C7A81E644	0x61807D991A7B6914	0x61807D991A7B6914	0x61807D991A7B6914	Yes

SQL Server saw that we had forced a Query Plan. It also recognized that the row estimates in the forced query plan were way lower than the estimates for the currently executing query, so it created a morally equivalent plan which has the same shape as the forced plan, but larger row estimates.

That’s cool, but this weird bug showed up, too:

The “morally equivalent plan” XML has the wrong query_hash value in it. It’s stored the query_plan_hash as the query_hash, too!
The query_plan_hash in the execution plan for plan_id = 11 does not match the query_hash in sys.query_store_query.

Here’s what this looks like when viewed in the query plan properties in SSMS:

It Gets Worse: Further Executions Make the Query_hash Incorrect for Every Row with This Query_id in Sys.query_store_query

Now run the stored procedure for another value:

EXECUTE dbo.PostsForDisplayname N'Kendra';
GO

Rerunning the same query to check query_hash values in the dynamic management views shows that we have a new “morally equivalent plan” added.

And the Query Store DMVs have now become infected with that messed up query_hash value.

I’ve extracted the data for query_id 7 to text, which is:

Query Id	Plan Id	Query Hash sys.query_store_query	Query Plan Hash sys.query_store_plan	Query Hash in Plan XML	Query Plan Hash in Plan XML	Is Use Plan Hint in Plan?
7	9	0x61807D991A7B6914	0x44A186BFC52AA053	0xD55EEC4C7A81E644	0x44A186BFC52AA053
7	10	0x61807D991A7B6914	0x56049465CF9FFB95	0xD55EEC4C7A81E644	0x56049465CF9FFB95
7	11	0x61807D991A7B6914	0x61807D991A7B6914	0x61807D991A7B6914	0x61807D991A7B6914	Yes
7	12	0x61807D991A7B6914	0x868D1CD7976DCFF8	0x868D1CD7976DCFF8	0x868D1CD7976DCFF8	Yes

Now we are in a state where:

One “morally equivalent” query plan has erroneously set its query_hash value to the query_plan_hash value in the plan XML
Another “morally eqivalent” plan being generated has caused the query_hash value in sys.query_store_query to be bizarrely updated to the value of the query_plan_hash for one of the plans.
We have three query plans where the query_hash value in the plan XML does not match the query_hash value in sys.query_store_query
If we have any tooling that has recorded the previous query_hash value of 0xD55EEC4C7A81E644 for this query, it (or you) will no longer find that in sys.query_store_query.

Update: I’m Not the First to Find This Bug, and I Probably Won’t be the Last

I was inspired to write this post after Haripriya Naidu asked at PASS Data Community Summit if anyone else had seen this behavior, and I had – I’d just been so burnt out on filing bugs with Microsoft that I hadn’t found the energy to document and report it. (It’s a lousy customer experience.) But the fact that Haripriya asked made me feel like it was worth it.

Paul White has written a great post digging into this, SQL Server Forced Plans Overwite the Query Hash. Paul shows a much simpler way to reproduce this than I have here, simply with a use plan hint. (I do think the repro in this post still adds some value by showing how the issue spreads in Query Store.)

Paul writes:

This bug affects SQL Server 2008 to 2022 CU16 inclusive.

He clarifies in a note:

Only guided plan search is affected. Supplying individual query hints (except USE PLAN of course) in a plan guide (or via an equivalent feature) does not provoke this misbehaviour.

Paul also references that this problem appeared in posts by SQL Sasquatch, aka @sql_handle back in 2019.

Tomáš Zíka also posted about this.

I have chatted with a Product Managers at Microsoft in a LinkedIn DM about this who works with the teams in this area. They were receptive to hearing about this issue– and they even started looking into it during their weekend. I do expect that working through issues like this can take some time for a product like SQL Server, and I’m happy that the first steps of finding the right people to communicate the issue to and raising awareness of the bug within the user community seem to be going well.

What This Means If You Force Query Plans

I’m hoping Microsoft recognizes this a bug and fixes the issue, because this makes using Query Store much harder for administrators. I think, most critically, it makes learning to use Query Store harder.

First of all, it makes no sense to me that the hashes of two different things (query text and a query plan) would regularly be the same value. Holy hash collision, Batman. It just is confusing.

But this also means:

You can’t reliably use the query_hash value from an execution plan that you find with a tool like sp_WhoIsActive or from a monitoring tool to find the query in sys.query_store_query.
- Using the query_plan_hash and looking it up in sys.query_store_plan should be more reliable, but you have to know to do that.
If you have a single tenant architecture (multiple copies of the same database schema in production), you can’t reliably find the same query across different query stores using the query_hash value.
- If query plans have been forced in some places, the query_hash could be set to who knows what, and you’ll incorrectly think the query isn’t running there.
- You can’t use query_plan_hash for this, either, because possibly the query has totally different plans.
- last_compile_batch_sql_handle in sys.query_store_query is potentially the best bet at this point. But that’s for the entire batch. Maybe statement_sql_handle in sys.query_store_query_text is the best bet, because that value is also in query plan xml.
The same thing also goes for queries that you’ve saved off in the past and want to examine now, but Query Store data has changed.

What Is the CPU Wait in Datadog SQL Server Monitoring? How Does It Compare to Waiting on CPU?

help@KendraLittle.com (Kendra Little) — Thu, 21 Nov 2024 00:00:00 +0000

I use Datadog on a regular basis, and I’m a pretty huge fan. The monitoring pack for SQL Server (and its PAAS variants) is still pretty rudimentary, but it evolves regularly. That’s NOT what I’m a fan of, though.

What makes me a raving fan is the flexibility of Datadog’s notebooks and dashboards, combined with the ability to create all sorts of custom metrics and monitors. There are always things in SQL Server monitoring packs that I have strong opinions about. Datadog lets me take what I want, build what I need that isn’t contained in that, and ignore the rest. For a team that has the budget to afford Datadog paired with dedicated database staff with the time and resources to do this work, this can be a great fit.

One of the weirdest and worst parts of the Datadog SQL Server monitoring tooling, though, is how it handles wait stats. In my opinion, it’s a case of someone reinventing a wheel that didn’t need to be reinvented, and then not documenting what they did clearly (at least not in a way I can find).

Two of the most confusing Datadog ‘waits’ are labeled ‘CPU" and “Waiting on CPU”. I opened a support ticket with Datadog a while back to ask what these are, because I couldn’t find any way they correspond to actual wait stats in SQL Server. I learned they aren’t wait stats at all. In fact, I think you should largely ignore them. Here’s why.

Waits in Datadog are All Mixed Up with Session Status

The most important thing I learned from the support ticket is that what are listed as “waits” in Datadog are purposefully not just the wait types we are familiar with from SQL Server. They have invented the concepts of “CPU” and “Waiting for CPU”, which are based on query/session status.

What determines the “CPU” and “Waiting for CPU” wait event is a combination of a couple of things:

The query’s current status (pulled from sys.dm_exec_requests (Transact-SQL)).

The session’s current status (pulled from sys.dm_exec_sessions (Transact-SQL)).

The query’s current wait type (pulled from sys.dm_os_wait_stats (Transact-SQL)).

The “CPU” Wait Isn’t Actually a Wait

Here’s an image of how Datadog displays so-called CPU “waits” – it’s really prominent:

When you look at these graphs, it’s almost impossible to not think this is some kind of representation of a CXPACKET wait or SOS_SCHEDULER_YIELD. However, it’s not that at all:

If the query doesn’t have a wait type and the request status is running or if the session status is running, then the Wait Event will be CPU.

It’s just a sample of the number of running queries.

Why in the world would that be included as a ‘wait’ and labeled ‘CPU’? I don’t know that anyone did any user research with DBAs here.

For workload/activity metrics, I prefer looking at something like batch requests/second. I don’t find this sample of executing queries at an instant useful, so I ignore it.

“Waiting on CPU” Isn’t a Wait, Either

I guess it’s a good thing that there’s a similarly named wait, “Waiting on CPU”, because at least that’s a clue that a “CPU” wait isn’t the same thing in some way? I donno. Here’s what “Waiting on CPU” looks like:

This so called “wait” represents a query that is in the runnable queue that isn’t running at the instant of the sample:

If the query doesn’t have a wait type, and the request status is runnable, then the Wait Event will be WAITING ON CPU.

I don’t think this is particularly useful, either. The SQLOS cycles queries on and off the runnable queue very, very frequently. What’s sampled at an instant like this isn’t super meaningful to me.

Datadog’s SQL Server Monitoring would be Better Off with a Much Simpler Approach to Wait Stats

My take: One of the benefits of SQL Server is that it has very strong documentation and an ecosystem of users who generate a lot of content about things like wait stats. Just show us the waits with their actual names, don’t rename them or decode them. Don’t make stuff up. That’s really all you need to do. Renaming things and making up “waits” to inject into graphs with confusing names only makes it harder for users to understand your tool.

But like I said– I love using Datadog. While I don’t find the way it displays SQL Server wait stats super useful, the flexibility of the tooling makes it OK for me to disagree with their take on that and still build observability tooling that does what I need by filtering for the waits I care about and setting up custom metrics that help me alert quickly on problematic situations– plus I can build in self-healing. And that, dear reader, is glorious.

Thanks very much to Datadog support for helping me decode this when the answer wasn’t clear at all.

How to Survive Opening a Microsoft Support Ticket for SQL Server or Azure SQL

help@KendraLittle.com (Kendra Little) — Mon, 23 Sep 2024 00:00:00 +0000

Asking Microsoft for support for SQL Server or Azure SQL is a lousy experience these days, whether you’re using a cheaper service tier or the more expensive support tier formerly known as “Premiere Support.” You need to know a lot about the root cause of your problem and how to solve it, or your request will be dismissed with misinformation. You’ll need data and metrics to back up your claims to get the ticket escalated, and you’ll need to provide those receipts multiple times. Once escalated to the Product Group, you may get a helpful response, but it takes a while. If the answer is relayed through a lower support tier, it often won’t make much sense.

These issues aren’t due to bad work ethics or personal failings of support workers. These are good humans trying their best. The problem is worse because it’s systemic.

A culture of guessing at answers

Support staff don’t seem equipped to learn about the products they support. There’s a culture of guessing at answers, which indicates they lack access to subject matter experts. They also have trouble accessing telemetry for hosted services. I suspect they can’t easily see previous messages or emails I’ve sent.

But you’ll still need help from Microsoft to report bugs or outages. The support experience is repetitive, time consuming, and frustrating. Here are my tips for making it through as efficiently as possible.

Start the ticket promptly, but immediately start researching on your own

You’ll have to be the expert here. Many support employees know just enough to be dangerously wrong.

With the cheaper support tier, you’ll need to persistently restate the exact product you’re using. You’ll get responses that only apply to a different SQL Server offering—queries that don’t run, answers from another universe.

With the more expensive tier, they understand what product you’re using, but knowledge about how it works is very limited. It’s like interacting with ChatGPT: you’ll get blocks of complex text that don’t hold up to scrutiny.

For example, if you report I/O errors in an Azure SQL Managed Instance error log, you might be told this is how General Purpose throttles storage and you need to pay for a more expensive tier. This could cost a lot, unless you understand that:

You can prove I/O usage was well below limits using sys.server_resource_stats
The error means exactly what it says: 15+ seconds of storage not responding, which is a storage subsystem failure
This isn’t how I/O throttling works on the service

🔥 UPDATE: Microsoft has announced the general availability of the Next-gen General Purpose service tier for Azure SQL Managed Instance, which includes improvements to I/O latency, IOPS, and transaction log throughput. This section describes errors common on General Purpose V1 blob storage. You don't want that.

You might also be told that RECOMPILE hints cause PAGELATCH waits, that you need to be the only active user to shrink a file, or that if a query triggers stack dumps you should just stop running it.

Support quality has degraded significantly in recent years, partly due to Azure SQL products having similar names, multiple rebrands, and blended documentation. It’s hard to learn about these products without real-world experience, and I’ve never seen evidence that support staff get that opportunity.

You may need a consultant

If you don’t have someone on staff who understands SQL Server, it’s worth hiring a consultant to help manage the support process.

I realize this feels like it shouldn’t be necessary. Microsoft Support should be better. But I have little hope it will improve soon—Generative AI will have the same problems. Early versions of Gen AI Advice in the Azure Portal already recommend solutions for the wrong product, just like the cheapest support tier.

Working with someone who knows SQL Server may seem expensive, but so is SQL Server licensing and cloud costs. A good expert will help you call out bad advice from Microsoft Support, which often suggests throwing more money at licenses or Azure to spend the problem away.

I don’t think there’s nefarious intent from support staff—they’re doing their best with the documentation they have, which generally puts an overly optimistic spin on more expensive tiers. But that advice often isn’t in your best financial interest and won’t work. There’s a conflict of interest when your cloud provider also creates the software: organizational incentives are around profit margins, not helping customers find cost-effective solutions.

Keep all your research in a document

You’ll need to provide data repeatedly. Don’t fall into the trap of only entering data into the email thread—it gets long, and people miss information.

I keep a copy of all my research and data in a document so I can quickly restate it without going back through the thread. I imagine support staff can’t see anything I provided previously and work as if they’re in Memento or Groundhog Day. I can’t change the system, but I can make it easier to manage.

This also makes it easy to share with the team: the problem, data gathered, ticket status, workarounds evaluated, and what we need from support.

Save blurbs that block common brush-offs

Support folks work off scripts. You’ll quickly notice patterns: if you report unexpected downtime in Azure SQL Database or Managed Instance, you’ll get long paragraphs about how your application needs retry logic, suggested as a “solution.”

If you already have retry logic (like almost every application), save phrases you can use to head these off proactively or redirect back to the real problem. It’s not worth writing these each time—you get to work off scripts, too.

Frequently ask for escalation to the Product Group

If you aren’t getting a helpful answer, ask early and often for escalation. Depending on your support contract, there may be several levels needed. Folks in the Product Group know the product well and can diagnose problems. They can fix real bugs and give useful advice.

Sometimes you can’t find the right people. For example, I think nobody is home when it comes to the Change Tracking feature and its intermittent problems with synchronous Availability Groups at high load.

But most of the time, if I can get an issue escalated with the right supporting data, I get helpful information back. It just takes a while, and I need to do research and testing on my own, plus come up with workarounds.

Someone’s going to call at the end

Microsoft will call you at the end when you’ve reached some sort of acceptance about your issue, even if you’ve specified you prefer email. I’m guessing metrics show that talking to a human lessens frustration. They’ll ask how your support experience was.

Like all the support people involved, this will be a good human trying their best. But this whole system shows little signs of positive change.

SQL Server Page Compression: Should You Worry About CPU Usage Increasing on Inserts, Updates, and Deletes?

help@KendraLittle.com (Kendra Little) — Sun, 08 Sep 2024 12:48:50 -0400

Every time I share a recommendation to use data compression in SQL Server to reduce physical IO and keep frequently accessed data pages in memory, I hear the same concern from multiple people: won’t this increase CPU usage for inserts, updates, and deletes?

DBAs have been trained to ask this question by many trainings and a lot of online content – I used to mention this as a tradeoff to think about, myself– but I’ve found this is simply the wrong question to ask.

In this post I’ll share the two questions that are valuable to ask for your workload.

Are Data Modifications Slow Enough That They are Impacting Application Performance/customer Experience, or Close to It?

If you already have a problem where data modifications are problematic and are impacting your customer’s experience, have a look at what is causing those problems.

In this section, I’m listing the issues that plague most systems and cause slow modifications. All of these make data modifications increase by seconds or minutes:

Poorly optimized modification queries – including using merge syntax: If modifications are slow, your first step is to find the query plans and execution statistics for the modification queries. Are they doing lots of reads? Are they having issues with parameter sniffing? Merge statements may save some lines in writing code, but they often create terribly performing execution plans. (Please read Michael J Swart’s post, “What to Avoid if You Want to Use MERGE for even more reasons to avoid merge.)
Cascading updates/deletes: These are a recipe for slowness when it comes to foreign keys in SQL Server. The cascading updates and deletes occur within the calling transaction scope, and the serializable isolation level is used to help ensure data correctness. If you have these in place and are using them, data compression is the last thing I’d worry about for slowness on modifications: you’ve already got a much bigger monster in the house.
Non-optimized or overloaded triggers: Not much explanation needed for this one, I think. Everyone hates digging into trigger code to optimize it, but sometimes you need to put on your hazmat suit and go in there. And sometimes you need to talk to other folks about what business logic can move into application code and not need to run in a trigger.
Pessimistic isolation levels increasing locking and blocking: Sometimes modifications are slow because they can’t get the locks needed to commit. SQL Server implements a pessimistic version of the Read Committed isolation level by default, under which readers block writers, and writers block readers. This can massively slow down modifications.
Non-indexed foreign key relationship: Let’s say we have two tables, Invoice and Payment. For every invoice, we might have 0 or more payments. There is a foreign key on Payment.InvoiceId that ensures that all InvoiceIds in Payment have a corresponding “parent” row in the Invoice table. The bit that most folks miss about this is that when an InvoiceId is updated or deleted in the Invoice table (perhaps for an archive process or a process synchronizing data between sources), the database engine has to make sure that this isn’t creating a problem in the foreign key relationship. If Payment.InvoiceId does not have a supporting index and and the Payment table is large, this can make that modification to Invoice very slow. Note: in this case, applying data compression to the structures would likely make the modifications faster, as it’s shrinking the structure that has to be scanned. But either way, the foreign key relationship needs a supporting index so it doesn’t have to do a scan to check referential integrity.
Indexed indexed view lacking appropriate supporting indexes: When you create an indexed view that joins between multiple tables, you need to make sure that it doesn’t adversely impact modifications to those tables. If you don’t have the appropriate indexes on the member tables to support the join columns, data modifications may cause scans of the tables. This is another case where compressing data structures on the member tables could actually speed up modifications by reducing the page count of what is scanned, but really what you need is good indexing.

If you are concerned about data modification performance, by all means evaluate what is causing your modifications to be slow and solve those problems as a priority!

But meanwhile DBAs have been trained to worry about data compression. Data compression is generally something that increases data modifications by a matter of milliseconds. (I say generally only because I’m sure there’s someone out there who has a horrific schema where data compression manages to increase modification time by a second or more – there’s always an exception.)

Does Data Compression Reduce Your Overall CPU Usage? This Matters Because It can Reduce Your Licensing Costs

This is the valuable question to ask about CPU – what is the CPU impact that data compression has on your workload overall? This is worth exploring because it can save you significant money.

I have seen data compression have an impact on CPU by lowering it for OLTP workloads. This happens when data compression helps reduce the number of physical reads that your system is doing, especially when storage is slow. Here’s how the magic works:

Data compression reduces the amount of data pages an index takes up, both on disk and in memory
This allows more data to fit into the data cache (buffer pool), which reduces the amount of physical reads for queries
Reads from memory are MUCH faster than reads from storage, so the amount of time queries take to execute speeds up
Faster queries mean fewer queries cycling on and off schedulers, there’s just more room for everyone – CPU usage drops

Have you ever noticed that tuning indexes can allow you to dramatically reduce CPU usage on a SQL Server. This works for the exact same reasons.

But Kendra, I have a Narrow Table That’s Modified a Million Times a Minute, can You Prove It Won’t Cause CPU Problems?

Not unless you’re paying me a consulting rate. Look, I’m not telling you to compress EVERY index or to not use common sense. I support you in looking for low hanging fruit, and there is almost always PLENTY of that to spend your time with.

Any Database Change can Cause a Regression

Making changes to databases is tough, I get it. SQL Server workloads are quite complex, and even simple “best practice” changes can cause unexpected regressions. I’ve even had adding memory and CPUs to a database instance cause regressions by changing query plans when the optimizer’s choice on the new plan wasn’t ideal.

Always be careful applying a change:

Use pre-production systems to validate changes.
Work within your change control process and document what you are deploying, and when.
Identify how to version control your changes, and understand how you will roll them back if needed.
Baseline performance before you make changes.
Deploy changes in discrete chunks.
Observe how your baseline adjusts as the change burns in.

But all that said, data compression in SQL Server is still an underrated and underused feature of Enterprise Edition, and concerns about CPU impact to data modifications unnecessarily block many people from benefiting from the feature. It can speed up many things you would think it might slow down, including even modifying data types.

Script to Automate Unforcing Failed Forced Plans in Query Store (SQL Server)

help@KendraLittle.com (Kendra Little) — Mon, 02 Sep 2024 11:35:40 -0400

tldr; I’ve published a script to loop through all databases on an instance, identify if there are any query plans in a problematic ‘failed" forced state (which can hurt query performance), and un-force them if found. Get the dbo.dba_QueryStoreUnforceFailed stored procedure on GitHub.

This script is designed to work on SQL Server on-prem, in a VM, or in Azure SQL Managed Instance or SQL Server RDS. Since the script is instance-level and loops through all databases, this isn’t really designed for Azure SQL Database – and you don’t get a SQL Agent there anyway, so you probably want to change this around for that use case. The script is shared under the MIT license, feel free to contribute code and/or adapt away for your own uses.

What’s So Bad About Failed Forced Plans?

Query plans that have been forced in Query Store may end up with the forcing in a “failed” state for a variety of reasons.

Sometimes this might be that a structure in the forced plan is no longer valid– maybe an index that had been selected in the plan has been dropped or its definition has changed. This plan will have a last_force_failure_reason of 8712 and a last_force_failure_reason_description of NO_INDEX in sys.query_store_plan.

I’ve found that if a forced plan is cancelled by the calling application while it is still compiling, it will have a last_force_failure_reason_description of GENERAL FAILURE, and that this scenario can cause serious performance problems by dramatically increasing compilation time while the query has the failed forced plan. See my post General Failure Failed Forced Plans in Query Store Cause Even Slower Compile Times for a script to reproduce this problem.

I do think there are likely more scenarios where failed forced plans cause slow compilation, other than the GENERAL FAILURE state. And failed forced plans do you no good at all– there is no benefit to leaving them in a failed forced state.

So if you force plans at all, whether manually or by using the Automatic Plan Correction feature, it is worthwhile to set up a process to check for failed forced plans and un-force them.

Get the Script

Get the dbo.dba_QueryStoreUnforceFailed stored procedure on GitHub.

Shouldn’t SQL Server Un-force These Itself?

I think so. Unfortunately, it does not. Add this to the “job security for database specialists” file, I suppose.

I’ll be Talking About This and More at Pass Community Data Summit in November

I’m giving a regular session, Perf Tuning in Prod: Plan Forcing, Plan Guides, and Plan Correction in Seattle at the PASS Community Data Summit in November.

You can save $150 off the 3 day conference registration with the code LITTLEK24. Register today.

Please Compress Your Indexes and Shrink Your Databases If You Use Azure SQL Managed Instance

help@KendraLittle.com (Kendra Little) — Mon, 26 Aug 2024 00:00:00 +0000

Shrinking databases in SQL Server isn’t fun – it’s slow, it causes blocking if you forget to use the WAIT_AT_LOW_PRIORITY option, and sometimes it persistently fails and refuses to budge until you restart the instance. You only want to shrink a SQL Server database when you’ve got a good reason and a lot of patience.

If you’re using Azure SQL Managed Instance and you haven’t already used data compression on your indexes and shrunk your databases, you probably have two good reasons to do both of those things: performance and cost reduction.

Use Page Compression on Your Data to Make the Most of Memory

Compressing indexes reduces the amount of data pages that indexes take up in storage. That reduces the amount of physical reads your queries do to pull that data into memory. That’s terrific, but it’s only half the benefit: compressing those data pages helps you fit more data into memory at a given time, which further reduces physical IO. Physical IO is your enemy in the video game that is administering Managed Instance.

You don’t get a ton of memory per vCore with Managed Instance:

Standard series hardware: 5.1 GB RAM per vCore. 16 vCore= 81.6GB RAM.
Premium series hardware: 7 GB RAM per vCore. 16 vCore = 112 GB RAM.
Premium series memory-optimized hardware: 13.6 GB RAM per vCore. 16 vCore=217GB RAM.

All of these amounts of RAM make me sad. That memory-optimized option doesn’t seem very memory-optimized to me. But the lower options are basically anti-patterns. To make a SQL Server workload perform well, you want to make sure you can keep frequently accessed data in cache. This is especially true when you have slow storage, which is the case with Managed Instance.

Yes, the storage is faster on the Business Critical tier, but at a huge cost— and even then, it ain’t all that fast and your writes will get gated by waits due to synchronous replicas.

If you can use a combination of compression, data pruning, and index tuning to make the General Purpose tier work for you, you will get a lot more value out of the service, without the synchronous commit waits. But you may have to shrink your databases, too, because…

Storage Sizes are Tied to Vcore Allocation on Managed Instance

My least favorite thing about Managed Instance is that the amount of storage you need on an instance can dictate the number of vCores you must allocate. This can really drive up your monthly costs if you have a significant amount of data that is not read often– and that’s the case for most databases that have been around for longer than a year or two.

Here are some storage limits based on vCores. These limits are for total space used across all system and user databases – including tempdb. The prices listed are at pay-as-you-go rates as of today in the East US region if you are using “memory-optimized” hardware:

General Purpose:

Need more than 2TB? 8 vCore minimum. $2,375.04/month for 8TB.
Need more than 8TB? 16 vCore minimum. $4,753.76 /month for 16TB.
Need more than 16TB? Sorry, you can’t have that. (You can have more cores, but this list is based on storage sizes.)

Business Critical:

Need more than 1TB? 8 vCore minimum. $3,383.12/month for 2TB.
Need more than 2TB? 16 vCore minimum. $6,774.24/month for 4TB.
Need more than 4TB? 24 vCore minimum. $10,037.36/month for 5.5TB.
Need more than 5.5TB? 32 vCore minimum. $25,364.16/month for 8TB.
Need more than 8TB? 48 vCore minimum. $38,051.04/month for 12TB.
Need more than 12TB? 64 vCore minimum. $50,737.92/month for 16TB.
Need more than 16TB? You can’t have that. But you could still pay $96,570.24/month for 16TB at 128vCore. Yikes.

🔥 NOTE: Business Critical SKUs with more than 24 vCores were not available for pricing at the time of this writing in the East US Region. I had to look in the Central region to get pricing on them. I have found the availability of larger vCore Business Critical instances to vary a lot in East US.

Don’t Shrink Too Far If You’re on General Purpose V1

General Purpose V1 is very odd: throughput for your files is determined by their size.

You never want to shrink your files too far, just to end up in a cycle of constant shrinking and growing. But you also don’t accidentally dip under some of these IOPS thresholds.

There is an additional limitation of “Max 120 MiB/s per instance” for transaction log files under GPV1, and I can confirm that I haven’t seen much of a difference in performance between a 130GB log file and a 514GB log file. (I haven’t been all that scientific about it, though.)

Thankfully, this whole “IOPS based on file size” thing is going away in General Purpose V2 (in Public Preview at this time). But it will always be useful to stay fresh on what limitations apply to your configuration, and what configurations are possible.

The Cloud is Different

With on-prem databases, it’s often not such a big deal to let “maybe someday we will need it” types of data collect in your database, sprawling in non-compressed fields of bytes. The cloud is different. With PAAS databases, it can take a bit of time to understand how the various SKU options impact your costs, and how to get the best deal of them.

To recap, when it comes to Azure SQL Managed Instance, you can get good performance even on General Purpose, but to get there you really need to make your active working set of data fit into memory as much as it possibly can. This list will help you get there:

Maximize your memory per core with the Memory-Optimized sku (even if it’s not all THAT optimized)
Tune your indexes to minimize reads for your frequent queries
Delete old data where you can, truncate tables you don’t need
Use page compression on All The Things!
Shrink your databases if the steps above leave you with significant empty space

These steps should at least help you lower that storage slider for some cost savings, but it can often help you fit into a smaller vCore count and save loads of money over time.

I do Like Managed Instance – and I’m Excited About the Future of the General Purpose Tier

While there are many things I want to change about Managed Instance, you can get good value from the service in the right configuration, especially if you work to minimize your costs.

Here are three things I love about Managed Instance:

It’s good at Availability Groups. Whether you are using Business Critical with synchronous AGs or asynchronous AGs in a Failover Group (or both), it really takes the headache out of managing those replicas. This is not an easy thing to automate. I’ve found a gotcha or two, but I’m super impressed with how well it usually works.
It’s not ALL Availability Groups. I love that the General Purpose tier in Managed Instance uses a Failover Cluster like architecture. I don’t want the overhead of synchronous Availability Group replicas unless there is a specific need for them. For many purposes, having a failover cluster architecture in the primary region and an asynchronous secondary using a Failover Group in another region is a terrific fit. This is not available in Amazon RDS for SQL Server.
It’s much faster than it used to be to create and scale instances. A lot of the pains there have been fixed –as long as you’ve got the 2022 feature wave applied to your subnets. Ask your account rep about upgrading if you’ve been using Managed Instance for a while and don’t have that, it makes a lot of things better.

I’ll expand on these in a future post after I look a bit more at General Purpose V2, which is in public preview.

Error 1119 When Shrinking Database: Removing IAM Page Failed

help@KendraLittle.com (Kendra Little) — Tue, 20 Aug 2024 19:32:07 -0400

At times when shrinking a data file in a SQL Server or Azure SQL Managed Instance/Database, shrink operations may persistently fail with the error:

Msg 1119, Level 16, State 1, Line 11
Removing IAM page ([filenumber]:[pagenumber]]) failed because someone else is using
the object that this IAM page belongs to. DBCC execution completed. If DBCC printed
error messages, contact your system administrator.

There’s not much documented on this error anywhere that I can find, so I’m sharing my experience with this error.

TLDR; I was not able to get past this without restarting the SQL Server service.

How to Get Past the Error

So far, the only way I’ve found to get past this error is by restarting the SQL Server service, or accomplishing this by triggering a failover in a managed database environment. I’ve asked Microsoft Support to document this error and if there are any fixes other than a service restart.

SHRINKFILE with NOTRUNCATE may be Helpful If You Can’t Restart the SQL Server Service Right Away

If you can’t restart the SQL Server service right away, I have found that if you can run the DBCC SHRINKFILE command with the NOTRUNCATE option without hitting the error, this can help you make progress on shrink while you wait for a service restart.

(To be clear, this won’t actually shrink any files– it will only relocate pages within the datafiles. This can make subsequent shrinks move more quickly once you can remove the NOTRUNCATE option from your command without hitting an error.)

Things That did Not Work

I was not able to identify what object the IAM page belonged to– I believe it was used in a system object and I didn’t have the right breadcrumb trail to identify it. There were no longrunning open transactions on the instance. Stopping things like Extended Events traces didn’t help. Running SHRINKFILE with various permutations didn’t help either– shrinking smaller sizes worked to a point, then everything (except shrinking with NOTRUNCATE) on the data files would stop with the error. The error also happened on multiple data files with different IAM page numbers.

Microsoft Support recommended running checkdb, which came back clean. They didn’t actually recommend the service restart.

General Failure Failed Forced Plans in Query Store Cause Even Slower Compile Times

help@KendraLittle.com (Kendra Little) — Mon, 12 Aug 2024 00:00:00 +0000

This post demonstrates two related bugs with plan forcing in Query Store which increase the likelihood of slower query execution and application timeouts in SQL Server environments.

These bugs are most likely to impact you if:

You use the Automatic Plan Correction feature in SQL Server, which automatically forces query plans.
Anyone manually forces query plans with Query Store.
You have slow storage, which can increase your likelihood of having longer compilation times.

The General Purpose tier of Azure SQL Managed Instance and Azure SQL Database feature both slow storage and Automatic Plan Correction enabled by default. So, weirdly enough, your risks of suffering from this problem are high if you are an Azure SQL customer.

Thanks to Erik Darling for his help in diagnosing and reproducing these issues– and his ‘slow compiler’ query used in this post was incredibly helpful to isolate and narrow down these problems.

The Two Bugs

If a query has a forced plan in query store and the query is cancelled by the calling user/application while the query is still compiling (which does still happen for queries with forced plans), it will:
- Generate a query_store_plan_forcing_failed event without a clear error message (Error: 316, Severity:-1).
- The query will have the count of failures incremented in sys.query_store_plan
- The last_force_failure_reason_desc column will be set to GENERAL_FAILURE.
Following this, the query will have slower or potentially infinite compile time whenever it needs to recompile.
- The query used here to reproduce this issue normally takes 28 seconds to compile on my laptop.
- When in a GENERAL_FAILURE state it takes more than an hour to compile. (I got tired of waiting and cancelled it at that point.)

General failure is about right.

We can Reproduce These Problems with an Empty Database

This demo does some things that you wouldn’t want to do on a production system, use a test system instead. I reproduced this issue on Microsoft SQL Server 2022 (RTM-CU7) (KB5028743) - 16.0.4065.3 (X64)

First, create a database with Query Store enabled:

USE master;
GO

IF EXISTS
(
 SELECT
 1/0
 FROM sys.databases AS d
 WHERE d.name = N'QueryStoreTest'
)
BEGIN
 ALTER DATABASE QueryStoreTest SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
 DROP DATABASE QueryStoreTest;
END;

CREATE DATABASE QueryStoreTest;
GO

ALTER DATABASE QueryStoreTest SET QUERY_STORE = ON;
GO

/* These options for testing, NOT prod */
ALTER DATABASE QueryStoreTest SET QUERY_STORE
(
 OPERATION_MODE = READ_WRITE,
 MAX_STORAGE_SIZE_MB = 1000,
 INTERVAL_LENGTH_MINUTES = 5,
 SIZE_BASED_CLEANUP_MODE = AUTO,
 WAIT_STATS_CAPTURE_MODE = ON
);
GO

USE QueryStoreTest;
GO

ALTER DATABASE CURRENT SET COMPATIBILITY_LEVEL = 160;
GO

Start the “Slow Compiler” Example Query in Session A

Start this query in one session, let’s call it Session A. Thanks to Erik Darling for this portable example of a slow-compiling query.

SET STATISTICS TIME, IO ON;
GO

WITH val as
 (SELECT * FROM (VALUES ( 1 , 2), ( 1 , 2)) as v(i,j) ),   
 p0 AS (SELECT pt.* FROM val AS pt),
    p1 AS (SELECT pp.* FROM p0 AS p CROSS JOIN p0 AS pp),
    p2 AS (SELECT pp.* FROM p1 AS p CROSS JOIN p1 AS pp),
    p3 AS (SELECT pp.* FROM p2 AS p CROSS JOIN p2 AS pp),
    p4 AS (SELECT pp.* FROM p3 AS p CROSS JOIN p3 AS pp),
    p5 AS (SELECT pp.* FROM p4 AS p CROSS JOIN p4 AS pp),
    p6 AS (SELECT pp.* FROM p5 AS p CROSS JOIN p5 AS pp),
    p7 AS (SELECT pp.* FROM p6 AS p CROSS JOIN p6 AS pp),
    p8 AS (SELECT pp.* FROM p7 AS p CROSS JOIN p7 AS pp),
    p9 AS (SELECT pp.* FROM p8 AS p CROSS JOIN p8 AS pp)
SELECT    c = COUNT_BIG(*)FROM p9 AS p;
GO

Watch Compilation Time with sys.dm_tran_active_transactions in Session B

Monitor how long it takes to compile by running this query in another session, let’s call it Session B.. I’m currently seeing around 28 seconds on my laptop. The first query doesn’t need to complete execution, it only needs to complete compiling. Cancel the query after it finishes compiling– you’ll know that happens when the query no longer shows up as a row in the query in Session B..

SELECT
 transaction_begin_time = at.transaction_begin_time,
 seconds = DATEDIFF(SECOND, at.transaction_begin_time, SYSDATETIME()),
 at.*
FROM sys.dm_tran_active_transactions AS at
WHERE at.name = N'sqlsource_transform';
GO

This images shows Session B when the query from Session A had been compiling for 26 seconds:

Shortly after this point, it finishes compiling and no longer shows up as a result of this query.

Find the Query Id and Plan Info in Query Store and Force the Plan

In session B, run this query to find the query_id and plan_id for the long compiler. (I’ll refer to this as the ‘Query Store Status Query’ in following steps.)

SELECT
 query_id = q.query_id,
 plan_id = p.plan_id,
 query_sql_text = qt.query_sql_text,
 avg_compile_sec = p.avg_compile_duration / 1000000.,
 last_compile_sec = p.last_compile_duration / 1000000.,
 is_forced_plan = p.is_forced_plan,
 force_failure_count = p.force_failure_count,
 plan_forcing_type_desc = p.plan_forcing_type_desc,
 last_force_failure_reason_desc = p.last_force_failure_reason_desc
FROM sys.query_store_query AS q
JOIN sys.query_store_query_text AS qt
 ON q.query_text_id = qt.query_text_id
JOIN sys.query_store_plan AS p
 ON q.query_id = p.query_id
WHERE qt.query_sql_text LIKE N'%(SELECT * FROM (VALUES ( 1 , 2), ( 1 , 2)) AS v(i,j) ),%';
GO

In my case, this is query_id = 2 and plan_id = 2.

Run the following query in Session B to force the plan. Specify the correct query id and plan id that you identified in the query above:

EXECUTE sys.sp_query_store_force_plan
 @query_id = 2,
 @plan_id = 2;
GO

After forcing the plan, rerun the Query Store Status Query and you should see that the plan has been forced. There haven’t been any failures so far.

Optional: Configure an Extended Events Trace

If you would like, you can now create and start an Extended Events session looking for the query_store_plan_forcing_failed event. It will fire when you do this next step.

Clear the Procedure Cache, Then Restart the Long Compiler Query and Cancel It Immediately

We need to do something to cause the query to recompile. For real world queries, this might be an automatic statistics update, a failover, or anything else that triggers a fresh compilation.

After doing this sequence of events, rerun the Query Store Metadata Query to review the status now:

forced_failure_count = 1 last_force_failure_reason_desc = GENERAL_FAILURE

Restart the Long Compiler Query Above for the Third Time: It’s Even Slower

Monitor its compilation time in Session B using sys.dm_tran_active_transactions to look for ‘sqlsource_transform` as before.

It will compile for a very long time – more than 1 hour on my laptop.

Here it is at 8 minutes, still compiling:

And here it is just past an hour, still compiling:

Reminder: these extra-long compile times will not be written to Query Store if the query is cancelled before compilation finishes.

This invisibility in Query Store makes identifying and diagnosing this issue even tricker than normal. Any external monitoring you have may report queries as having run and timed out over and over again, but when you look in Query Store, you don’t see the query even executing.

What should Microsoft do About This?

It feels to me like there are multiple bugs in here that Microsoft should fix:

A query that is cancelled during a compilation timeout does not seem like it should result in a GENERAL_FAILURE status in Query Store when a plan is forced. Applications can have timeouts set at all sorts of things.
Plans with GENERAL_FAILURE as a plan forcing status should not take longer to compile.

And I do not think Automatic Plan Correction should be enabled by default in Azure SQL Managed Instance or Azure SQL Database.

I have reported these issues to Microsoft via a support ticket.

What should Users do About This?

If you ever force plans— and especially if you use the Automatic Plan Correction feature, which frequently forces plans– you should create a SQL Server Agent Job that regularly looks for failed forced plans with a status of GENERAL_FAILURE and unforce them.

Consider turning off Automatic Plan Correction if you have it enabled.

Slow Storage Can Cause Slow Compilation Time in SQL Server

help@KendraLittle.com (Kendra Little) — Thu, 08 Aug 2024 00:00:00 +0000

Up till now, I’ve thought of compilation time in SQL Server as being dependent only on CPU resources– not something that requires fast storage to be speedy. But that’s not quite right.

Slow storage can result in periodic long compile time in SQL Server. And long compile time not only extends the runtime for the query, it can also result in blocking with waits for compile locks.

Thanks to Erik Darling for helping me figure this out, and explaining this all in his video, What Else Happens When Queries Try To Compile In SQL Server: COMPILE LOCKS!. For great details and demos, go watch that! I’ll be working through the topic with some simple flow charts here.

Synchronous Auto-update of Statistics and Compilation Time

flowchart LR id1["`Synchronous Statistics Updates`"] --> id2["`Long Compilation Time for a Query`"]

Let’s unpack this.

Statisics

Statistics are objects that store information describing the distribution of data in a column or index. Statistics help the query optimizer identify efficient ways to execute a query by providing row estimates, which influences the selection of indexes and physical join types.
Databases in SQL Server default to automatically refreshing statistics when the data in a table changes significantly. This is great, it helps the query optimizer have a better chance at creating fast execution plans without human intervention.

There are two modes for auto-update of statistics in SQL Server

Synchronous stats updates (default): The query that triggers the statistics update waits for the update to complete before executing. This ensures the query uses the most up-to-date statistics but can cause a delay in execution. The most common misunderstanding here is that the query triggering statistics update will be a query that modifies the data. In fact, it is a query that needs to read from the table and use the statistics in optimization that triggers the auto update of statistics.
Asynchronous Stats Updates: The query proceeds without waiting for the statistics to be updated. The stats are updated in the background
1. This avoids delays for the query which triggers the automatic statistics update, but may result in the query using outdated statistics. This may increase the chances of that query getting an inefficient query plan.

Here’s what I didn’t realize

I had previously thought that under the synchronous mode of auto-update statistics, queries would wait to start compiling until the statistics were updated. In fact, that period of time waiting for statistics to update is counted as part of compilation time. (You can see this in Erik’s demo.) There is a wait statistic, WAIT_ON_SYNC_STATISTICS_REFRESH, which…

Occurs when waiting for synchronous statistics update to complete before query compilation and execution can resume.

Source: sys.dm_os_wait_stats docs

That description does say that compilation is effectively paused / has started when this wait happens. However, this wait is not surfaced in actual execution plans.

It is also not tracked in Query Store:

Query Store tracks wait stats only during query execution, not during query compilation. This restricts Query Store’s ability to track compilation wait stats.

Source: sys.query_store_wait_stats

In other words, long compilation times that you see in query store may or may not be long because of a wait on synchronous stats update OR a wait on compilation locks. You can’t tell from just query store. (Reminder: if your queries have a timeout while still in the compilation phase, they will NOT be recorded in query store for that execution at all.)

Compile Lock Waits and Blocking

flowchart LR id1["`Synchronous Statistics Updates`"] --> id2["`Long Compilation Time for a Query`"]--> id3["`Query Holds Compile Lock`"] --> id4["`Other sessions running the same query are blocked`"]

When you have slow storage and synchronous stats auto-updates, you are more likely to have long compilation times (because of what we described above), and you are therefore more likely to see compilation lock waits. This is especially true in an OLTP workload where some queries are run very frequently from multiple sessions, and are likely to run concurrently.

Compile locks in SQL Server prevent changes to the plan cache during query compilation to ensure consistency. If we run the exact same procedure or parameterized query from multiple sessions, and one is having a long compile time, the others wait until it has compiled and cached the plan. They can then pull this plan from the cache and off they go.

This is great when compilation is fast. But when compilation time starts taking 20 seconds, 30 seconds, or longer, this can become quite disruptive.

How do You Fix This?

I think the easiest thing to monitor for is the presence of WAIT_ON_SYNC_STATISTICS_REFRESH waits. If you don’t have those waits, long compilation time is not related to waiting on synchronous stats updates.

If you do see those waits, look for blocking on compilation locks. This can help you identify which queries are impacted by this. If it’s only a few queries, you may want to work at simplifying those queries.

If it’s not just a small set of queries, consider changing your database setting and using asynchronous statistics updates on the database. There’s a few disclaimers when it comes to async stats updates. As mentioned above, this setting may result in the query which triggers automatic stats update using outdated statistics. This may increase the chances of that query getting an inefficient query plan.

But additionally, Microsoft warns:

Asynchronous statistics update is performed by a background request. When the request is ready to write updated statistics to the database, it attempts to acquire a schema modification lock on the statistics metadata object. If a different session is already holding a lock on the same object, asynchronous statistics update is blocked until the schema modification lock can be acquired. Similarly, sessions that need to acquire a schema stability (Sch-S) lock on the statistics metadata object to compile a query can be blocked by the asynchronous statistics update background session, which is already holding or waiting to acquire the schema modification lock. Therefore, for workloads with very frequent query compilations and frequent statistics updates, using asynchronous statistics can increase the likelihood of concurrency issues due to lock blocking.

Source: Statistics docs

There is a setting that can help with this if you are running SQL Server 2002, Azure SQL Database, or Azure SQL Managed Instance:

In Azure SQL Database, Azure SQL Managed Instance, and beginning in SQL Server 2022 (16.x), you can avoid potential concurrency issues using asynchronous statistics update if you enable the ASYNC_STATS_UPDATE_WAIT_AT_LOW_PRIORITY database-scoped configuration. With this configuration enabled, the background request will wait to acquire the schema modification (Sch-M) lock and persist the updated statistics on a separate low-priority queue, allowing other requests to continue compiling queries with existing statistics. Once no other session is holding a lock on the statistics metadata object, the background request will acquire its schema modification lock and update statistics.

Source: Statistics docs

There are more details about this setting in Dimitri Furman’s blog post, Improving concurrency of asynchronous statistics update.

Consider testing those settings together if you’ve got the option.

The Restore-AzSqlInstanceDatabase Long Running Operation Failed Error on Azure SQL Managed Instance

help@KendraLittle.com (Kendra Little) — Thu, 30 May 2024 07:56:08 -0400

What’s it like to be a Database Administrator for managed databases in Azure? Sometimes it’s a painful guessing game when a routine, core operation– restoring a database – fails with a most unhelpful error.

In this case, if the restore is run via PowerShell, following Microsoft guidance, the error message is:

Restore-AzSqlInstanceDatabase: Long running operation failed with status ‘Failed’.
Additional Info: An unexpected error occured while processing the request.

The misspelling of “occurred” doesn’t bring confidence, but we try and try again.

Surely There’s a More Clear Error Message If You Run the Restore in the Azure Portal, Right?

Nope, if you attempt do to the same operation in the Portal it will fail, and the message there is:

Error code: ResourceOperationFailure
Message: The resource operation completed with terminal provisioning state 'Failed'.

Ooof. So what went wrong?

I Couldn’t Find the Answers in the SQL Server Error Log

One of the benefits of Azure SQL Managed Instance is that it behaves a lot like a ’normal’ SQL Server Instance. You can access the SQL Server Error Log.

But even for core operations like a database restore, it doesn’t always contain the info you’d expect. In this case I couldn’t find any info regarding the restore in the log at all.

I Couldn’t Find the Answers on the Internet, but I Found Another Customer Who Also Lost Hours to This Problem

I haven’t found anything in Microsoft documentation so far explaining what can cause The Vaguest Error Ever.

I did find a blog post from Mika Sutinen in 2021, SQL Server Managed Instance and the most unhelpful error message during a database restore. Mika hit the same error message as I did – and in Mika’s case it was worse because they were restoring a large database, so the “long running” part of the error implied it might be related to the database’s size.

In Mika’s case, the database was partially contained and they needed to run some sp_configure commands to get the restore to work. (See the link above for full details.)

Talk about a needle in a haystack.

But the database I was restoring wasn’t partially contained. I was looking for a different needle.

In My Case, It was a Transparent Data Encryption Problem

When using Transparent Data Encryption (TDE):

Backup files for databases that have TDE enabled are also encrypted with the DEK. As a result, when you restore these backups, the certificate that protects the DEK must be available. Microsoft Docs

If you hit issues restoring a TDE database in “normal” SQL Server and this isn’t configured correctly, you’ll receive error messages that give you a good clue – something along the lines of:

Cannot find server certificate with thumbprint....

I didn’t have access to any such message when restoring my database. And, in fact, I didn’t figure it out at all– my colleague Mike figured it out and ended my suffering after a couple of hours. (Thanks, Mike!)

Better Error Messages do Show Up in Some Places

Azure SQL Managed Instance also has a database cloning functionality that uses an AG-like setup under the hood, called Database Copy.

This process seems to use some sort of database restore to seed the AG. If initiating the copy fails, you get a more clear error message.

This isn’t great as a workaround, though, because there are a lot of limitations on database copy, the biggest of which is:

The source or destination managed instance shouldn’t be configured with a failover group (geo-disaster recovery) setup.

If you have to choose between maintaining Disaster Recovery and troubleshooting a restore failure, most folks are going to choose keeping DR healthy every time.

Can Someone Please Fix This?

I created an “idea” on the Azure feedback site, Database Restores in the portal and Restore-AzSqlInstanceDatabase should give a helpful error message.

Erik Darling and Kendra Little Rate SQL Server Perf Tuning Techniques

help@KendraLittle.com (Kendra Little) — Mon, 20 May 2024 10:12:32 -0400

Erik Darling joins Kendra Little to rate different SQL Server Performance Tuning Techniques in episode 81 of the Dear SQL DBA podcast. We share our opinions of… (deep breath)

Recompile hints, Query Store hints and plan forcing, CTEs, Resource Governor, the legacy cardinality estimator, Table Variables, Automatic Plan Correction, Batch Mode, index rebuilds, Hekaton, NOLOCK, page compression, partitioning, filtered indexes, columnstore, join hints, PSPO, indexed hints, indexed views, optimize for unknown, RCSI, adding more memory, restarting the damn thing, scalar UDFs, and Persisted Memory Grant Feedback.

Listen to the Podcast

Watch on Youtube

More Dear SQL DBA

Check out the episode list.

Finally, a SQL Server Monitoring System That Leverages Query Store

help@KendraLittle.com (Kendra Little) — Wed, 15 May 2024 19:14:39 -0400

I’ve spent a bit of time with Microsoft’s new database watcher tool for Azure SQL recently.

There are a lot of things I like about database watcher– which is currently in preview and which refuses to Capitalize Its Name– but it does one big thing that I really, really like: it collects data from Query Store. You can access that Query Store data from built-in database watcher dashboards, query it using KQL, or (something something) in Microsoft Fabric if you’ve got money to burn on your monitoring data.

Query Store has been available since SQL Server 2016, but I haven’t yet heard of monitoring tools that truly take advantage of it. It’s about time.

Three Major Sources to Query Performance Data from SQL Server

Most monitoring tools for SQL Server collect fairly similar information:

What queries are running at a given time?
What were their basic metrics like duration, cpu time, logical reads, number of writes, query workspace memory used?
Were the queries waiting on resources for a significant amount of time, like lock waits?
What query execution plan was in use?

This data can be gathered from SQL Server in a variety of ways, which have different tradeoffs. Most major monitoring tools for SQL Server use a combination of Plan Cache and Query Stats DMV Sampling as well as some tracing to collect information:

APPROACH	PROS	CONS	AVAILABILITY
Plan Cache and Query Stats DMV Sampling	Most lightweight polling.	Limited or missing data for queries with recompile hints, will miss some data whenever plan cache is cleared	Since SQL Server 2005
XEvents or SQL Trace	Misses few events, high accuracy	Heaviest performance impact of the options. Some things, like execution plans, will really slow performance if collected this way.	Since SQL Server 2005
Query Store	Data is kept updated by SQL Server itself and is the basis for many recent SQL Server intelligent query processing (IQP) features	May not work well with default settings for some workloads, especially if allocated minimal space. Only aggregate performance metrics over specified collection interval. Misses queries that are cancelled before they finish compiling.	Since SQL Server 2016

Why Isn’t Trace Data or Plan Cache Data Enough?

Plan cache and trace data are certainly useful. Query Store still only collects data from writable primary instances, so for readable secondaries they are the only options we’ve got. (Query Store for Secondary Replicas, a sort-of-maybe feature from SQL Server 2022, remains in preview here in May 2024. It is not available even to preview in Azure SQL Database or Managed Instance. It does feel like a lot might be wrong with it, the longer it lingers.)

But Query Store data is, well, it’s just more reliable. When I use a monitoring tool that polls the SQL Server Plan Cache for a lot of its information (which is most of the time these days), I often find that in real world troubleshooting situations that data is just not quite right. Some things are missing. Some things will be confusing. This is increasingly true for performance issues that slow the whole system down – external polling for monitoring information is impacted by this as well, and it may completely fail for the most problematic periods.

I find that in these cases, Query Store contains the information I need to figure out what was going on. Query Store doesn’t have complete information – the monitoring from plan cache is still useful. But when I want a ‘source of truth’ about query execution plans and aggregate performance metrics, Query Store data is what I want to look at. And if there were major waits — from tempdb metadata waits to resource_semaphore waits to threadpool waits — Query Store gathered the data when not much else worked.

However, I don’t want to/ can’t maintain Query Store data in every database forever. I don’t want it to grow so large that it hits sized based cleanup.

And I also don’t love querying production to get performance data, either. I’d rather it go into a monitoring system.

Why Aren’t All Monitoring Companies Doing This for SQL Server?

I haven’t heard of other SQL Server monitoring tools leveraging Query Store. I did some searching on this to see if this had happened and I hadn’t noticed, and found a set of very funny forum posts where users ask if a SQL Server monitoring tool is going to use Query Store, and a sales rep responds with a version of, ‘Hey, we collect all this query data! Look at it! Here’s a picture! Please don’t notice that I’m not answering your question."

I don’t think any of the major SQL Server monitoring tools will likely treat Query Store as a major source of data collection for a while, for largely timing reasons.

Most of these products have been established for a while, and they already built a foundation on using the plan cache/ DMVs with some trace data sprinkled in.
Enterprise sales depend on a tool being able to cover many versions of SQL Server, and lots of folks still run SQL Servers older than 2016.
Query Store isn’t enabled by default in SQL Server databases until SQL Server 2022, and even then it’s only for new databases created – it’s not enabled for existing databases on upgrades automatically. There are plenty of SQL Servers where Query Store isn’t running.
Query Store does need to be sized appropriately to avoid cleanup problems or it often slipping into read-only mode, when it doesn’t collect data. The plan cache and trace approaches don’t have this weakness.
See my above rude commentary on Query Store for readable secondaries growing old while still stuck in preview.
A very high number of users don’t need super accurate Query Store data – they aren’t specialists. They need simpler, more high level metrics to act on.

Given all of this, I don’t really blame monitoring companies for not rewriting their tooling to rely on Query Store immediately.

However, I still want Query Store data to be collected into a datastore. I’m part of the portion of specialist users who need this data to ensure I’m diagnosing a situation accurately and recommending the right course of action. Isn’t it about time we got something in this area?

Why This Works for Database Watcher

Database watcher currently supports Azure SQL Managed Instance and Azure SQL Database. Query Store is available to all users of these services, and it’s enabled by default when you create a new database.

While there have been a variety of monitoring-ish tools for Azure SQL over the years, none of them have gotten much traction. They’ve all been so similarly named that I never learned how to tell them apart, to be honest. But basically, there was no “this has been around for 10 years”, robust, widely adopted SQL Server engine monitoring system to displace in Azure.

A fresh start was made, and when you’re making a fresh start with a tool specifically for “current” Azure SQL Engine versions, using Query Store as a major data source is a clearer choice.

For me, it’s a very refreshing one. I’ve already found data in database watcher that has helped solve tough problems.

One final note: having a monitoring system collect data from Query Store helps you balance Query Store size and data retention settings– if the data is being collected by database monitor, you don’t need to retain nearly as much history in the production database. That makes managing Query Store size easier, and things work better all around.

Hopefully other monitoring solutions add Query Store in as a data source before another 8 years go by.

Learn Perf Tuning in 2 Days at PASS Summit 2024 With Erik Darling and Kendra Little

help@KendraLittle.com (Kendra Little) — Wed, 01 May 2024 08:15:30 -0400

I’m teaming up with Erik Darling to teach you SQL Server Performance Tuning in two days at the PASS Data Community Summit in Seattle.

Erik and I are co-teaching both days of training to give you a strong strategic background on the internals you need to know, along with critical tactical performance tuning techniques. Join us to level up your perf tuning skills!

About Erik Darling and Kendra Little

Erik Darling is the founder of Erik Darling Data. Erik is a top consultant who helps customers around the world troubleshoot SQL Server performance problems and write reliably fast T-SQL, against all odds, every day.

I’m Kendra Little. These days I’m a Staff Database Reliability Engineer in cannabis tech, a rapidly growing industry with super high performance requirements. I help developers write and tune TSQL using Azure SQL, observability tooling, and lots of automation.

Together, Erik and I bring a road-tested, proven understanding of SQL Server internals: we teach you what you need to know to get the job done without wandering too far into the weeds of confusion. We’ll share which techniques really work, what features are mostly hype, and how to solve a tricky problem without your clever solution falling apart at the seams.

Day 1 - Practical Guide to Performance Tuning Internals

Whether you’re aiming to be the next great query tuning wizard or you simply need to tackle tough business problems at work, you need to understand what makes a workload run fast– and especially what makes it run slowly.

Erik Darling and Kendra Little will show you the practical way forward, and will introduce you to the internal subsystems of SQL Server with a practical guide to their capabilities, weaknesses, and most importantly what you need to know to troubleshoot them as a developer or DBA. They’ll teach you how to use your understanding of the database engine, the storage engine, and the query optimizer to analyze problems and identify what is a nothingburger best practice and what changes will pay off with measurable improvements.

With a blend of bad jokes, expertise, and proven strategies, Erik and Kendra will set you up with practical skills and a clear understanding of how to apply these lessons to see immediate improvements in your own environments.

Day 2 - Query Quest: Conquer SQL Server Performance Monsters

Picture this: a day crammed with fun, fascinating demonstrations for SQL Server and Azure SQL. This isn’t your typical training day; this session follows the mantra of “learning by doing,” with a good dose of the unexpected.

Think of this as a SQL Server video game, where Erik and Kendra guide you through levels of weird query monsters and performance tuning obstacles. By the time we reach the final boss, you’ll have developed an appetite for exploring the unknown and leveled up your confidence to tackle even the most daunting of database dilemmas. It’s SQL Server, but not as you know it—more fun, more fascinating, and more scalable than you thought possible.

Q: is Day 1 Required to Understand Day 2? does Day 2 Repeat Material?

Nope! These two days are designed to compliment one another without repeating material: each precon is a force multiplier for the other, but each day also can stand alone and be fully understandable.

Join Us in Seattle on November 4 and 5 at Pass Data Community Summit!

Hope to see you there :)

Query Store Size Based Cleanup Causes Performance Problems - How to Avoid It

help@KendraLittle.com (Kendra Little) — Tue, 23 Apr 2024 07:47:55 -0400

I’m a huge fan of SQL Server’s Query Store feature. Query Store collects query execution plans and aggregate query performance metrics, including wait stats. Having Query Store enabled makes troubleshooting performance issues such as bad parameter sniffing, much, much easier. Because Query Store is integrated into SQL Server itself, it also can catch query plans in a lightweight way that an external monitoring system will often miss.

When performance matters, it’s important to ensure that you’re managing Query Store so that Query Store cleanup does not run during high volume times. Query Store cleanup could slow your workload down significantly.

Look Out, the Cleanup Hornets are Coming

Cleanup isn’t easy. I know this from having needed to delete or update data in various circumstances over the years – it’s tough to clean things up safely without impacting performance.

Query Store has some issues with this currently as well.

If you collect wait stats in Query Store, Query Store size-based cleanup of wait stats runs a query:

DELETE
 pws
FROM sys.plan_persist_wait_stats AS pws
WHERE pws.plan_id IN
(
 SELECT
 pp.plan_id
 FROM sys.plan_persist_plan AS pp
 WHERE pp.query_id = @query_id
);

If you monitor your SQL Server or Azure SQL database from a third party monitoring system like Datadog or Redgate SQL Monitor, you will be able to see this query running in that system. You’ll also be able to see it in the plan cache.
When Query Store has a good amount of data– but not necessarily enormous (say, 4GB), this query can run repeatedly for a very long time when called by cleanup.
This query has significant impacts– more than I would have expected. In an environment with a synchronous availability group in a high activity period, this can depress the rate of transaction log flushes and notably raise the level of HADR_SYNC_COMMIT and HADR_GROUP_COMMIT waits.

I can produce this behavior/impact in a test environment with a synchronous Availability Group setup using Ostress.exe, after first building up a volume of data in Query Store.

I have not been able to produce similar impacts from other parts of Query Store cleanup, only wait stats cleanup, but it might be that they exist and my testing simply didn’t uncover them. Better safe than sorry.

When to Consider Not Collecting Wait Stats in Query Store

If you have a performance critical database and you want to ensure this doesn’t happen, you don’t have to collect wait statistics in Query Store.

I find the wait stats in Query Store to be occasionally useful only. They are aggregate wait stats, so they can give a general idea for simple issues like whether or not a query often waits on lock waits, but they aren’t super granular. If you do have an external monitoring system that is collecting wait stats, and if it allows you to correlate those wait stats to queries, you might simply decide not to collect wait stats in Query Store.

Notes:

Modifying settings in Query Store will clear your execution plan cache, so don’t do this at a peak time if that’s an issue for you.
If you’ve been collecting wait stats in Query Store and you disable collection, it does not immediately truncate the wait stats. Cleanup will still need to take care of them, unless you clear Query Store.
- Obviously, that will cause you to lose Query Store data. You might mitigate this by restoring a backup of the database to access that data if you need it, or you might be willing to start with a clean Query Store, up to you.

Size Based Cleanup - Avoid This

Query Store has two types of cleanup: time based and size based.

If performance is critical, you should tune your query store so that only time-based cleanup runs.

Things to know about size based cleanup:

It is based on MAX_STORAGE_SIZE_MB when SIZE_BASED_CLEANUP_MODE is set to AUTO. “Size-based cleanup will be automatically activated when size on disk reaches 90 percent of max_storage_size_mb. This is the default configuration value. Size-based cleanup removes the least expensive and oldest queries first. It stops when approximately 80 percent of max_storage_size_mb is reached.” Docs
If you’re working to reproduce issues with sized based issues, you may hit scenarios where your Query Store goes into read-only mode before cleanup starts. I found that I could avoid this by doing a little dance: I ran a workload to grow Query Store to 90% of MAX_STORAGE_SIZE_MB. Then I stopped my workload briefly (cleanup seems to need a lull) and ran sp_query_store_flush_db. Then I started my stress test again immediately. The timing is a bit delicate, but this allowed me to run a test load at the same time size-based cleanup was running.

You may want to allow Query Store some more room in MAX_STORAGE_SIZE_MB so that sized-based cleanup doesn’t kick in. Be aware that you shouldn’t go completely wild with this – as the fantastic Erin Stellato mentions in Query Store Best Practices: MAX_STORAGE_SIZE_MB should be “set to 10GB at the absolute max, something less ideally”

Erin recommends a custom capture policy to help if you need to reduce the amount of space that Query Store data takes up.

Erin mentions in Remove Data from Query Store that…

ideally, size-based cleanup never kicks in…. This algorithm is not efficient, and it runs single-threaded. It looks for queries that are infrequently executed/less important and deletes those, one by one, until the size is less than 80% of MAX_STORAGE_SIZE_MB. Avoid this type of cleanup if at all possible.

My testing shows this to be very good advice.

To monitor size based cleanup, you can trace the query_store_size_retention_cleanup_started and query_store_size_retention_cleanup_finished Extended Events. Read more in Zikato’s answer on StackOverflow.

The Mysteries of Time Based Cleanup

Josh Darnell has written an excellent post on Timing of the Automatic Query Store Purge. I won’t rehash his whole post here– go read it!

The most important basic bits are:

Time based cleanup happens every 24 hours
It’s roughly based on startup time

That means that if you happen to have a failover in the middle of the day, you may end up with time-based cleanup running in the middle of the day. Not ideal. Depending on your tolerance of cleanup running at a high volume time, this might be worth a planned failover at a low volume time.

As much as I don’t love that, I still find Query Store to be useful enough in general to deal with that.

To monitor time based cleanup, you can trace two Extended Events: query_store_db_cleanup__started and query_store_db_cleanup__finished. Only time-based cleanup events are captured in these events.

DIY Cleanup

There are a few stored procedures you can do to add custom cleanup jobs to run on schedule if you like. Erin Stellato give examples of them in ‘Remove Data from Query Store’:

EXECUTE sys.sp_query_store_remove_query
 @query_id = ZZZ;

EXECUTE sys.sp_query_store_remove_plan
 @plan_id = YYY;

EXECUTE sys.sp_query_store_reset_exec_stats
 @plan_id = YYY;

Product Improvements That would Help

The following things would be super helpful:

Allow configuration of when time-based cleanup runs
Give more insight into which bits of Query Store are taking up space and what levers to configure to “fit” Query Store into your desired maximum size.

And one correction… I had suggested:

Increase visibility of when size-based cleanup runs. I would really prefer for this to be logged in a DMV rather than an extended event, as that’s easier to get into monitoring systems, but I’ll take whatever I can get.

Update: this already exists as an Extended Event. See Zikato’s answer on using query_store_size_retention_cleanup_started and query_store_size_retention_cleanup_finished on StackOverflow.

Happy performance tuning, and please remember to tip your Query Store (by not allowing size-based cleanup to run).

Thanks Very Much to Erin Stellato, Josh Darnell, and Tomáš Zíka

Erin is super helpful, so smart, and just one of the best folks around. All of her posts on Query Store are incredibly helpful, just search “Erin Stellato Query Store” when you have a question :) If you work with Erin, please tell her that her awesomeness is widely loved.

Thanks also to Josh Darnell for his excellent posts on Query Store Cleanup. This was also super helpful to me. And thanks to Tomáš Zíka (Zikato) for his answer detailing how to trace size-based cleanup.

Top 5 Toxic Flavors of Tech Execs

help@KendraLittle.com (Kendra Little) — Thu, 11 Apr 2024 00:00:00 +0000

I’ve been an employee at small, medium, and large companies, and I’ve also been a short-term consultant working with a new company in any given week. I’ve worked with hundreds of tech companies remotely and have visited companies onsite on multiple continents.

Air-dropping into company cultures to work through problems with teams and present recommendations to their leadership reveals common anti-patterns in leadership – plus some patterns that make teams raving fans of their management.

Here’s the top 5 Toxic Flavors of Tech Execs I’ve encountered over 20 years, plus the Top 5 Team Building Tech Execs I’ve found, too.

Five Toxic Flavors of Tech Execs

As a consultant, teams have sometimes brought me in to help them make headway with these styles of leaders. It’s not that I have magical powers, it’s simply that when you bring in an outside expert on a topic, often a leader will listen to their voice differently. And short term consultants have an easier time speaking truth to power– they’ve got much less at risk than long term employees.

5. The Clumsy Penny Pincher

These execs are highly focused on reducing cost and increasing efficiency, but they’re also short sighted. They may not carefully consider the contracts they sign, so they get the worst end of the deal. Their teams struggle with a pile of perverse incentives – like having a service desk whose success metrics are based on closing tickets, but not actually resolving problems. Cost reduction can go well, but it can also bring productivity to a grinding halt. It’s tricky to make recommendations that anyone will have time to carry out: scope, scope, scope.

4. The Auto-Responder

These leaders have an established way of handling difficult conversations: they shut it down with a favorite catch phrase. They use the catch phrase enough to start a meme. Catch phrases are risky when they replace exploring problems: they communicate that the leader isn’t willing or able to listen. It is what it is. There’s no need to reinvent the wheel. Let’s just move forward. We need to think more positively. Let’s put a pin in this and move on.

3. Unlimited Waffles

“We had too many Priority 1’s so we created Priority 0. Now we have too many Priority 0’s.” Every day is a new adventure with these leaders, either chasing a new rainbow or running from yesterday’s nightmare. Just like Groundhog Day, you don’t make it far before starting over. It’s tricky to make improvements in this org because you can only plan for incredibly short cycles, as few things get momentum to last longer than a few weeks. If you’re in this team, you’ve gotta always think small.

2. The Hero Worshipper

These leaders invented the 10x developer. They celebrate employees who sacrifice their personal lives. But they only celebrate a hero briefly, then they’re on to their next best friend. This leader creates teams who are willing to (or need to) suffer loss of sleep and be absent from their families and friends for the sake of visa support or salary. Collaboration isn’t really a thing around the hero worshipper. Most people are stressed, all the time. Short term consultants can be temporarily effective here as they can break up the established dynamic, but it’s hard to create lasting change.

1. The Really Angry One

We still have folks in tech who lead with overt displays of power. These senior leaders humiliate and dress down employees in medium and large meetings, and often don’t allow the employee to respond or speak further. That employee sometimes disappears in a few days; no announcement is made for their departure. At other times, these execs may be warm and effusive, but they’re mercurial. Few leaders intend to be perceived this way; most agree it’s better to be loved than feared. But some gamble that it’s better to be feared than hated. Some panic often and express that in displays of power. This is the environment which is toughest to make a positive impact in: a command-and-control culture laced with fear is quite resistant to change.

My Top 5 Team Building Tech Execs

Leading organizations is a big challenge, I’m not saying otherwise. I am in awe of people who are great managers and executives in tech. They do exist, in both large and small companies.

Here are my favorite Talented Leader Types:

5. The Agile Nerd

Some of y’all will read this title and think “no no no, that’s an anti-pattern! The Agile-Will-Fix-Everything-Type goes on the other list!”

The leader I’m talking about here isn’t the Quick-Fix type. This is the leader who is deeply interested in finding ways of working that are flexible and which empower teams to do their best work with a customer-centric approach. They are open to criticism, love iterative improvement, and they find joy in learning from everyone. This is often a leader who has a lot of humility, but rarely speaks about humility. They are simply mostly interested in learning, growing, and improving their teams and themselves.

4. The Curious Pragmatist

This leader brings a genuine curiosity to work. They are curious about technical problems and they are also curious about people. This curiosity is paired with pragmatism: they are curious and open to change, but they also focus on how to carry through important projects and maintain “stable-enough” priorities and room for employees to grow and change over time. This leader has the ability to share the larger organizational context with their teams in a way that encourages innovation, and they are also excellent at advocating upwards for their teams.

3. The Deep Listener

Some leaders have a strong willingness and desire to practice listening. They practice this with both senior and junior employees. They don’t always try to offer solutions to problems, but they don’t brush off problems, either. They have a way of connecting to people and building bonds and trust. They think a lot before they ask questions or make suggestions, and they listen and pay attention to how those questions are answered, and if they create discomfort. When these leaders find the right organization and team that matches their skills, employees express themselves more openly in both small and large groups, and people enjoy problem-solving together. Thoughtfulness has a way of spreading, and these are fun teams to work in.

2. The One Who’s Been There

Some leaders have strong empathy through years of experience working in the same roles in which their employees work. These folks often don’t have a fancy degree, or a college degree at all. They started small, and they still empathize and identify with the employees in the company who have the least amount of power. These leaders are effective at getting things done and are good at finding a middle ground without alienating people. They are often not a “big name” at work, but they are broadly respected. They have a calming presence, they are very easy for a wide variety of people to feel comfortable talking with, and they often have minimal turnover in their teams. When they move roles, their employees want to follow them.

1. The Force Multiplier

My favorite leadership style is the leader whose superpower is finding the superpower in their own employees and teams. This leader isn’t afraid of giving feedback, but they don’t focus on finding criticism. Instead, they reframe situations and identify how the strengths in their teams can be used to their best advantage. They are talented at thinking of ways to maximize those strengths and leverage uniqueness. These execs recognize potential where others don’t. These folks change lives.

I’m Generalizing

Leaders are people, too. I’m not going to go full “won’t someone sympathize with the CEO who laid thousands of people off,” but being a good leader takes practice, thoughtfulness, and dedication over time. Nobody is 100% a “bad” or “good” leader. Great leaders have made plenty of mistakes, learned from them, and improved, just like great individual contributors. I’m no expert at this– it’s easier said than done– but I am inspired by the great leaders I’ve worked with and I learn more every day. The anti-pattern archetypes that we all know well make good leadership even more important.

Something Weird Happened: I Love My Job Again

help@KendraLittle.com (Kendra Little) — Tue, 26 Mar 2024 08:32:04 -0400

Back in my 20’s, I was lucky enough to go to graduate school. I had a work-study job in the Dean’s office where I got to develop and administer their Access databases, which helped me get by.

One day, the Dean said to me: ‘Kendra, when you talk about your work on our databases, you light up. When you talk about your coursework, that doesn’t happen. Have you thought about that?’

That observation changed my life.

I finished up my Master’s degree and left graduate school. I got my first job in tech. I couldn’t land a job specializing in databases right away, but I kept getting more and more database experience and I eventually landed a job working with SQL Server full time.

I loved it.

Eventually, I Decided to Try Other Things

After working with databases for more than ten years, I felt like I should try new things. Not completely unrelated things— I still was very interested in database performance and software development– but I wanted to explore a bit.

I tried my hand at being a Developer Advocate, then a Product Manager.

I worked as a Content Developer, then as a Technical Product Manager for a Data Platform team.

I was good at these roles, and I learned a ton. As a TPM I worked with a terrific team who taught me a ton about Data Platforms and Data Science, which was fascinating. But for various reasons, I ended up leaving that role and spending a few months thinking about what I wanted to do next. It was tough to find an answer.

Six months ago, I saw a job listing for a Database Reliability Engineer at Dutchie that looked… well it looked amazing. It was a performance tuning role using PAAS databases in the cloud, focusing on SQL Server but with opportunities to also work with Postgres, which I’ve wanted to work with for a while.

I got lucky and landed the job.

I Ended Up Working Last Sunday

When you do production work with a 24 hour online system, you end up working occasional nights or weekends. (Automation and careful system design can lessen this a lot, but sometimes it’s just easier and faster to do something at a low volume time. And I could take Monday off.)

At one point on Sunday, my partner looked in my doorway and smiled. I don’t know what I was doing, probably laughing at the query optimizer’s choices. I had no idea it would sometimes use a key lookup in an index creation statement.

“You’re SO happy,” he said. “I love that you’re so happy now.”

Yeah, I was happy working on a weekend. Working with relational databases is what lights me up, after all these years.

I am still that kid learning the lesson that there’s something about databases which my brain can’t get enough of.

You Will Not Find Long Compilers Who Time Out in Query Store

help@KendraLittle.com (Kendra Little) — Tue, 05 Mar 2024 08:40:57 -0500

Last November, a puzzle was really bothering me. Some queries from an application were timing out frequently after running for 30 seconds, but they were halfway invisible in the SQL Server.

I say ‘halfway invisible’ because I could see the queries while they were running in SQL Server’s dynamic management views using free tools (sp_WhoIsActive and sp_BlitzWho).

But the queries had some odd characteristics:

They had a sql_handle, but never a query_hash
They also never showed a plan_hash or execution plan
I couldn’t find the timeouts in Query Store like a normal timeout

Query Store Won’t See a Query That Doesn’t Finish Compiling

This all happened around the time of the PASS Data Community Summit in Seattle. I thought, surely if I ask enough people, someone will have an idea that sticks.

I ended up in the Expo Hall, babbling about half invisible queries to anyone who would listen. This was actually a good strategy.

Erik Darling and a group of fine Microsoft PMs listened to my tale of confusion. Erik suggested, “Maybe the query isn’t done compiling?” Erin Stellato thought it through and confirmed that yes, if the query did not finish the compilation process before the client cancelled it, then that run of the query wouldn’t be recorded in Query Store.

(After this chat, I eventually found and read the Learn article on How Query Store Collects Data, which gives more detail on Query Store and how it fits into query processing. Read that to learn more, it’s good.)

Identifying a Long Compiler

Before this, I hadn’t really thought about how a query appears in SQL Server DMVs if it is still compiling. I’d taken short compile times for granted. It turns out I shouldn’t.

As I mentioned above, some traits of a query in the compilation phase are that you’ll see a sql_handle, but not a query hash, query plan, or query plan hash.

There is another way you can monitor these and confirm that a query is compiling as well: you can query sys.dm_tran_active_transactions for transactions where the name column has the value sqlsource_transform.

SELECT
 transaction_id = at.transaction_id,
 transaction_begin_time = at.transaction_begin_time,
 seconds = DATEDIFF(SECOND, at.transaction_begin_time, SYSDATETIME())
FROM sys.dm_tran_active_transactions AS at
WHERE at.name = N'sqlsource_transform';

Thanks to the Troubleshoot queries that seem to never end in SQL Server article for this little tip, it helped me confirm that I was, in fact, seeing queries that were taking longer than 30 seconds to compile, and being cancelled due to a timeout before they ever got a query plan.

Why were they compiling so long? I’ve got some details, but some of it is still mysterious. Stay tuned for more in a future post.

And many thanks to Erik Darling, Erin Stellato, and others for their invaluable help and advice.

How to Start an XEvents Trace on a Read Scale-Out Azure SQL Managed Instance

help@KendraLittle.com (Kendra Little) — Thu, 29 Feb 2024 08:21:15 -0500

It took me more than half hour to figure out how to start an XEvents trace on a read-scale out instance of Azure SQL Managed Instance. It’s hard to monitor read scale-out instances, so tracing is desirable! I started with a simple trace of sql_statement_completed. Hopefully this saves other folks some time.

Architecture Review: What are Read Scale-out Instances in Managed Instance?

Read scale-out instances are provided when you use the Business Critical tier for Azure SQL Managed Instance. These use an availability-groups like technology to keep databases fresh in a read-only format.

You can offload queries to this read-only replica with near realtime data. However, the instances are tricky to monitor: most 3rd party monitoring systems have no idea what these are, and they don’t emit performance counters like a ’normal’ instance.

If you have configured a Failover Group and the failover partner is in the Business Critical service tier, that partner will also have read scale-out instances.

Want to Use an Event File Target? Buckle Up

I typically like to create both an event file target and a ring buffer target if the trace may run for a while.

There’s a few steps to set up a Managed Instance to work with event file targets. You can reuse this configuration for multiple XEvents traces, if that’s a comfort.

Follow the steps in Create an event session with an event_file target in Azure Storage to do the following in the Azure Portal:

Identify or create the Azure Storage Account you want to use
Create a container/bucket in the storage account where your trace files will be written
Create a SAS token for the container

After this, the article instructs you to run a TSQL command against your Managed Instance to store the SAS token in a server-scoped credential.

Where to run this step: The Managed Instance which is the “base” of the read scale-out instance. This might be your primary writable replica, or it might be the failover partner, or it might be both. (It is NOT the read scale-out instance. Connect with ApplicationIntent=ReadWrite.)

If you aren’t sure if this may have been done before, check if a credential has been set up in your Managed Instance with the following query:

USE master;
GO

SELECT
 *
FROM sys.credentials AS c;
GO

The credential name will be something like:

https://exampleaccount4xe.blob.core.windows.net/xe-example-container

The identity will be SHARED ACCESS SIGNATURE.

Create the Trace Definition on the Primary Replica or Failover Partner – Ssms will Get You Halfway There

SQL Server Management Studio (SSMS) only partially supports creating XEvents sessions. You can use the built in wizard to configure the session, but when you go to configure where to write the event file target it will thing the url for the location is an error.

So you need to script out the definition to TSQL, put the URL into the TSQL, and execute it in SQL to create the XEvents trace.

Is this a stack of workarounds in a trench coat? Absolutely.

Make Sure the Trace Definition Syncs to the Read Scale-out

Your trace now should exist on the Managed Instance where you created it (a primary replica or a failover partner).

If you now connect to your read scale-out instance (ApplicationIntent=READONLY), you should see in Object Explorer that the definition has synced over.

I had to open a second instance of SSMS to get Object Explorer to have connections to both an instance and its read scale-out partner at the same time.

Start the Trace on the Primary Replica or Failover Partner, Then on the Read Scale-out Instance

At the time I wrote this, the documentation claimed:

An extended event session on a read-only replica that is based on a session definition from the primary replica can be started and stopped independently of the session on the primary replica.

I didn’t find that to be true. If I attempted to start the XEvents trace on the read scale-out replica at this point, I got the error:

Msg 3906, Level 16, State 2, Line 1
Failed to update database "master" because the database is read-only.

In order to get the trace started on the read scale-out instance, I had to do a little dance:

Start the trace on the primary replica or failover partner
Start the trace on the read scale-out partner

I put in a PR to update the docs, so hopefully someone verifies this and figures out if it’s the same or different for Azure SQL Database.

Stop the Trace on the Primary Replica or Failover Partner

If you don’t need the trace to run on the primary replica or failover partner, you can stop it there. But don’t drop it– you need to leave it there as long as you want to run it on the read scale-out instance.

Hopefully nobody comes along and cleans it up.

Verify Your Trace is Working

Is it writing to the targets and picking up data? After all this, I sure hope so.

The good news is that you can now open the results of event-file traces inside SSMS, like a normal XEvents trace. This isa recent improvement which is VERY nice– you used to have to download the files from Azure Storage, which required stopping a trace and other bother. It’s much more convenient to work with trace data in SSMS now for Azure SQL Managed Instance.

Should You Use SQL Server Readable Secondaries If Queries Can Fail Repeatedly at Any Time?

help@KendraLittle.com (Kendra Little) — Wed, 21 Feb 2024 00:00:00 +0000

If you use readable secondaries in Availability Groups or Read-Scale out instances in Azure SQL Managed Instance, you may have queries fail repeatedly if there is a glitch and statistics are not successfully ‘refreshed’ on the secondary replica. Those queries may keep failing until you manually intervene.

It’s unclear if Microsoft will ever fix this. There is a well established support deflection article which documents the issue and provides ‘workarounds’.

As a user on StackExchange wrote in September of 2022, “This is an outstanding bug in SQL Server which has been ignored for quite some time, unfortunately.” Based on a link provided in that post, it appears feedback was provided for this as far back as SQL Server 2012, when AGs were introduced.

More than 12 years later, how viable are readable secondaries / scale-out read servers for production use if it’s acceptable for queries to fail at an undocumented/unknown rate?

What Causes Statistics Not to Refresh on a Secondary? can You Prevent It?

It’s not really clear to me what causes this error. Microsoft’s support deflection article says:

This issue occurs because an active transaction prevents the cache invalidation log record from accessing and refreshing the statistics on the secondary replica.

When I ran into this issue, there were no long-running transactions on the primary replica or any secondary replicas. Potentially this had happened at some point in the past and the process that refreshes statistics gave up, or was a victim in a deadlock and didn’t retry? All I have is guesses.

The Workarounds Suck Here, Folks

If this issue happens to you, the article presents three options for workarounds:

Dump all of the caches on the secondary replica

I don’t love this step as a fix for a production database server. It’s fine on some workloads, but there are plenty where it causes performance issues. The article also indicates that this is not as “permanent” an option as option 3, too.

Fail the node over

OK, so this option is to take a brief outage on the server? Really? How is this option 2? Is this a production database or an Easy Bake Oven?

Manually delete the statistic on the primary and recreate it

I guess I choose this as the “least worst” option. For undisclosed reasons, this is labeled a “more permanent workaround.”

Do I want to automate proactively finding this issue if it happens and repairing it? No.

Does Microsoft Know How Often This Happens?

I don’t think Microsoft understands how often this happens.

First, the article linked above is intended to drive down support requests. This saves money for Microsoft, and frankly users appreciate it because none of us want to work through support tickets, however it means that Microsoft doesn’t get signals about how many people are hitting the problem.

I can say that this does still happen with the latest versions of SQL Server and that it occurs on Azure SQL Managed Instance, based on what I have seen and what I see others reporting in blogs and forum posts. The support article also references Azure SQL Database, so presumably it happens there as well.

What’s Up with the Mention of Sp_table_statistics2_rowset?

In Microsoft’s support deflection article, the full sample error is:

Msg 2767, Level 16, State 1, Procedure sp_table_statistics2_rowset, Line [Batch Start Line ] Could not locate statistics ‘’ in the system catalogs.

The sp_table_statistics2_rowset system procedure is not fully documented. This procedure is run automatically when you use a linked server to run a query, but the article doesn’t mention linked servers at all or say that the bug only occurs when using a linked server.

Since the article explains that a statistic is not present/not usable on the secondary replica itself, presumably this can impact queries running without linked servers, but it’s not clear to me if they fail or if they just ignore the statistic (and maybe get a worse plan). I have seen that a query may fail due to this when run with a linked server, but the same query may run successfully if run against the replica correctly – more on that in an upcoming post.

Can You Rely on an Architecture on Readable Secondaries with SQL Server / Scale-out Read Instances with Azure SQL Managed Instance?

Given how long this issue has been around and the level of customer impact it seems to have (queries failing repeatedly with no warning until an administrator intervenes), I think this is an important question worth asking.

I would love to hear more from Microsoft about this in their blog posts or support articles:

Why is the “manual” statistics fix “more permanent”?
Can this happen with statistics related to indexes?
What happens when sp_table_statistics2_rowset isn’t involved?
Is it something Microsoft intends to fix eventually? Who needs to complain to make that happen?

As it is, it just seems like another reason not to use readable secondaries.

Ugly Bug: SQL Server Online Index Rebuild Sometimes Happens Offline Without Warning

help@KendraLittle.com (Kendra Little) — Wed, 07 Feb 2024 00:00:00 +0000

🔥 UPDATE: This issue has now been documented. A note has been added to the Perform index operations online documentation page, stating: "Index rebuild commands might hold exclusive locks on clustered indexes after a large object column is dropped from a table, even when performed online."

I found a nasty bug in SQL Server and Azure SQL Managed Instance recently: sometimes an ‘online’ index rebuild of a disk-based rowstore clustered index (basically a normal, everyday table) isn’t actually ‘online". In fact, it’s very OFFLINE, and it blocks both read and write queries against the table for long periods.

If you manage to make it through a rebuild successfully, the problem goes away for future rebuilds of that clustered index – likely leaving you bruised and bewildered.

Fine Print for the ‘Well Actually’ Crowd

This bug isn’t related to the schema modification lock (SCH_M) taken out briefly at the end of an online index operation. That’s documented, and risks of that can be mitigated with the WAIT_AT_LOW_PRIORITY setting. This post describes and has a repro script for exclusive (X) locks held for long periods through the duration of the supposedly online operation. The WAIT_AT_LOW_PRIORITY setting offers no protection for these locks in my testing. I am also only discussing examples that drop newer types of LOB columns – my repro script uses NVARCHAR(max), and the problem also happens with VARBINARY(max).

When does This Happen?

I can reproduce this problem when rebuilding a clustered index online after dropping a large object column (LOB) from the table. The problem happens even if DBCC CLEANTABLE has been run to clean up LOB table from removed columns already.

What should happen is that the ALTER INDEX ... REBUILD WITH (ONLINE=ON) command should return an error if the operation can’t be performed online. And there should be a published way to detect or understand when online index operations aren’t available. but this isn’t the case. Instead, the supposedly “online” command runs very slowly and takes out exclusive locks. Depending on your environment, that might cause some big old problems.

If It Worked Properly, What would Happen?

This diagram from the How online index operations work article is helpful in understanding the locks that should be taken out in an online index operation for a disk based rowstore table:

To reiterate, this is Microsoft’s diagram from How online index operations work, and is not my own work.

Note that there aren’t any exclusive (X) locks in this diagram.

Why is This So Bad?

As Microsoft’s documentation says:

We recommend performing online index operations for business environments that operate 24 hours a day, seven days a week, in which the need for concurrent user activity during index operations is vital.

Online index rebuilds are only available in Expensive editions of SQL Server: Enterprise and Azure SQL. In my personal experience, online index rebuilds are one of the major features that sell licenses for those Expensive editions – while REORGANIZE commands can be performed online, they are fairly limited and are generally quite slow.

A bug in a major feature that sells those Expensive licenses is a pretty big deal. It’s not uncommon to use large object columns, and it’s also not uncommon to want to drop those columns once the data becomes unwieldy.

I’m quite surprised that this problem appears to have existed in SQL Server since 2020 and is reported to be a known issue by Microsoft Support since May 2021. I haven’t found this documented by Microsoft. I would expect this limitation to be listed in Guidelines for online index operations, Perform index operations online, and How online index operations work, but I can’t find a mention anywhere.

It is possible that the bug was fixed before and has recurred. Or it’s documented someplace I haven’t found yet. But it seems like a sad situation.

This Happens in SQL Server 2022, Azure SQL Managed Instance, and Probably the Rest of the Gang as Well

I’ve reproduced this problem in two environments: SQL Server 2022 Developer Edition and Azure SQL Managed Instance. The Stack Exchange post above mentions Azure SQL Database having the problem, but I haven’t verified that myself.

Feel free to use my script below as the basis to check out other versions of SQL Server if you’d like.

Does This Happen Under X, Y, or Z Scenarios?

I’ve been able to reproduce this problem with the script below. I haven’t tested a bunch of variations of the scripts with different table schemas, data types, etc, simply because time is scarce. I’m writing this at 9 pm and I still have raccoon drawings to color and shade before the post is done. (Priorities.)

Basically, this could happen only under the scenario I have described below, or it could also happen under other scenarios as well. Feel free to use the script as a start for your own testing and explore more.

How I Reproduced the Problem, with Screenshots

Setup: download and install the free diagnostic stored procedure, sp_WhoIsActive.

Next, create a little test database if you want. I used compat level 160 because i wanted to use GENERATE_SERIES() later on. I’ve seen this in compat level 150 as well.

CREATE DATABASE DropLOBColumnTest;
GO

USE DropLOBColumnTest
GO

SET NOCOUNT ON;
GO

ALTER DATABASE DropLOBColumnTest SET COMPATIBILITY_LEVEL=160;
GO

1. Create a table, dbo.LOBTest

I am making a questionable choice and using FILLFACTOR=1. The bug can absolutely happen with a fillfactor of 100%. I am lazy and I wanted a way to create a bunch of in-row data pages without actually having to calculate a bunch of data to fill them in, so I did that. Follow your heart and change it if you like.

DROP TABLE IF EXISTS dbo.LOBTest;
GO

CREATE TABLE dbo.LOBTest (
 i integer identity NOT NULL,
 j char(5000) NULL,
 k nvarchar(MAX),
 l nvarchar(MAX),
 CONSTRAINT cpk_lob PRIMARY KEY CLUSTERED (i) WITH (FILLFACTOR=1)
);
GO

2. Add some data to dbo.LOBTest

How much data depends on what type of database you are using and how patient you are. On Azure SQL Managed Instance, I used “GO 10” and could reproduce the issue. On my laptop I went ahead and did 50 rounds because it’s much faster.

INSERT dbo.LOBTest (j,k,l)
SELECT 1, REPLICATE('a',4001), REPLICATE('a',4001)
FROM GENERATE_SERIES(1, 10000) AS g
--GO 10
GO 50

3. Check how much data is in dbo.LOBTest

This table is measured by weight, not by volume.

SELECT
 t.NAME AS TableName,
 CASE i.index_id WHEN 0 THEN 'heap' ELSE 'clustered' END AS TableType,
 i.index_id,
 a.type_desc,
 CAST(SUM(a.total_pages) * 8. / 1024./1024. AS numeric(10,1)) AS SizeGB,
 SUM(p.rows) AS [Rows]
FROM
 sys.tables t
INNER JOIN
 sys.indexes i ON t.OBJECT_ID = i.object_id
INNER JOIN
 sys.partitions p ON i.object_id = p.OBJECT_ID AND i.index_id = p.index_id
INNER JOIN
 sys.allocation_units a ON p.partition_id = a.container_id
WHERE
 t.NAME = 'LOBTest'
 AND i.index_id=1
 AND a.total_pages > 0
GROUP BY
 t.Name, i.index_id, a.type_desc
ORDER BY SizeGB desc;
GO

I have 7.6GB of LOB data and 3.8GB of in row data.

3. Drop one of the LOB columns from dbo.LOBTest

Goodbye, column k:

ALTER TABLE dbo.LOBTest DROP COLUMN k;
GO

4. Check how much LOB data you have now (spoiler: it’s the same)

Run the query above to check how much data is in the table. You should see the same amount. When you drop a LOB column, the space isn’t deallocated.

There are two ways to get the space back: DBCC CLEANTABLE or a clustered index rebuild. (I’ve tested DBCC CLEANTABLE and it doesn’t fix this issue, but feel free to reproduce that if you want.)

Even if you don’t want to reclaim the space, eventually your index may become fragmented enough that it qualifies for a rebuild in a maintenance job.

5. Create and start an XEvents trace

We are creating this trace for two reasons:

A. The trace is going to slow our instance down a lot because tracing locks is slloooowwww. So don’t do this on an instance where you care about performance. It’s actually handy for this to be slow because it gives you time to start up a read query that will be blocked and inspect the situation with sp_WhoIsActive without having to rush too much.

B. The trace information is somewhat useful. We could filter it more, but see A.

First, SELECT @@SPID to find the session ID you’re going to use to rebuild the index. Then specify that session id in the trace definition.

SELECT @@SPID;

CREATE EVENT SESSION [Locks] ON SERVER
ADD EVENT sqlserver.lock_acquired(
 ACTION(package0.process_id,sqlserver.database_name,sqlserver.sql_text,sqlserver.tsql_stack)
 WHERE ([sqlserver].[session_id]=(52))),
ADD EVENT sqlserver.lock_released(
 ACTION(package0.process_id,sqlserver.database_name,sqlserver.sql_text,sqlserver.tsql_stack)
 WHERE ([sqlserver].[session_id]=(52)))
ADD TARGET package0.ring_buffer
WITH (MAX_MEMORY=8192 KB,EVENT_RETENTION_MODE=NO_EVENT_LOSS,MAX_DISPATCH_LATENCY=5 SECONDS,MAX_EVENT_SIZE=0 KB,MEMORY_PARTITION_MODE=PER_CPU,TRACK_CAUSALITY=ON,STARTUP_STATE=OFF);
GO

ALTER EVENT SESSION Locks ON SERVER STATE=START;
GO

6. Rebuild that index “online”

Now run:

ALTER INDEX cpk_lob ON dbo.LOBTest REBUILD WITH (ONLINE=ON);
GO

7. Try to query the table in another session

In another session, run:

SELECT COUNT(*) FROM dbo.LOBTest;

8. Look in horror upon your lock waits with sp_WhoIsActive

In a third session, run:

EXECUTE sp_WhoIsActive @get_locks=1;

Here’s what you should see: Your supposedly online index rebuild blocking your select query, which is waiting with a LCK_M_IS lock wait:

If you open the XML column for the locks for the ALTER INDEX statement, you should see that the ALTER INDEX statement has been granted an X (exclusive) object lock on dbo.LOBTest. There are also a bunch of exclusive page locks– in this case 233,345 of them.

9. What’s in that XEvents trace?

There’s a whole lot of locks in there, and among them you will see some X locks. Here’s the one on the object:

10. Stop and drop that trace

Goodbye, trace.

ALTER EVENT SESSION Locks ON SERVER STATE=STOP
GO

DROP EVENT SESSION Locks ON SERVER
GO

So, What to do When Online=sad?

You might wonder, what do you do if you have a very large table and you can’t take an outage window to rebuild the clustered index offline?

Yeah, I wonder that, too.

You can run DBCC CLEANTABLE to try to release some space.

And you can exclude the clustered index from your index maintenance jobs, or configure it to only have REORGANIZE operations, not REBUILDS. Ever. (I’ve only tested this briefly– the REORGANIZE operation does take out X locks, but in smaller/briefer batches and regularly releases them. I wouldn’t swear to that, though, it was just a quick test.)

Aside from that, I really don’t know.

Can you run a query to detect if any of your tables have had LOB columns dropped and are in a state where they might be prone to this issue? If you can, I haven’t figured out how yet.

Hopefully this will get fixed before another couple of years elapse – but I fear that note that was added to the documentation means it likely won’t.

Is the Azure SQL Managed Instance Business Critical Service Tier Worth the Cost?

help@KendraLittle.com (Kendra Little) — Mon, 29 Jan 2024 00:00:00 +0000

The Business Critical service tier in Azure SQL Managed Instance is a lot more expensive than General Purpose. For the extra money, you get a different architecture.

Is it worth the extra cost? Spoiler: your mileage will vary, but probably not. Let’s talk about why.

Business Critical Architecture

In the Business Critical service tier, all databases are on local SSD on each node (instead of user databases being on blob storage). Data is transferred between nodes via compressed transaction log stream in an Availability Group-ish format. I say ‘Availability Group-ish’ because the tech has clearly been customized quite a bit: it feels like SQL Server and Azure SQL Database had a baby and named it Managed Instance.

As a result of these differences, you can’t see information in the same DMVs as a standard Availability Group.

Read Scale-out Instances Feel Half-baked

You get one ‘read scale-out" instance with Business Critical at no extra charge, but it’s… well, it’s limited. It’s so limited that I’m not a huge fan.

While it’s true that this “provides 100% additional compute capacity at no extra charge to off-load read-only operations, such as analytical workloads, from the primary replica” – there’s some fine print.

You don’t get a normal connection string for a read scale-out instance. Instead, you connect to the primary replica with a connection property, ApplicationIntent=ReadOnly. The read scale-out instance doesn’t emit performance counters, either. These limitations mean you are unlikely to be able to connect 3rd party monitoring tools to a read scale-out instance unless the vendor has specifically designed for these limitations. You can query some DMVs.

Maybe there’s some sort of Azure monitoring you can rig up (which probably has a cost), but I can’t find any way to see the health of read scale-out instances in the Azure Portal by default. In fact, I can’t see anything about read scale-out instances in the portal at all. They’re pretty invisible.

As in any Availability Group (ish) scenario, long running queries on the read scale-out instance can block DDL changes from the primary replica. There is an automated process to terminate those to keep things from getting too out of sync. (This is the exact kind of scenario that makes me want robust monitoring so I understand the impact. But…. yeah.)

Instances for Failover

There are a couple/few failover partners kept on the side for high availability ("up to three secondary replicas (compute and storage) that contain copies of data"), with data synced to these constantly.

I’m not sure if there’s a guaranteed minimum number of replicas (I haven’t seen it documented), but there is clearly some kind of quorum involved/required to keep the primary online.

Architecture Drawing

Is the Cost of the Business Critical Architecture Worth It?

Here’s a pricing comparison that I collected a week ago when I wrote about the General Purpose Architecture. Using the East-US region as an example and assuming the instance needs to be on all the time for one year (8,760 hours):

Service Tier	Series	Hourly Rate	Annual Cost
General Purpose (16 vCore)	Standard Series	$4.035/hour	$35,346.60
General Purpose (16 vCore)	Premium Series Memory Optimized	$5.541/hour	$48,539.16
Business Critical (16 vCore)	Standard Series	$10.871/hour	$95,229.96
Business Critical (16 vCore)	Premium Series Memory Optimized	$13.883/hour	$121,615.08

For simplicity I listed Standard Series and Memory Optimized at 16vCores– use the pricing calculator if you want to see other options. These pricing examples don’t include Failover Groups— multiply by 2 to estimate that.

On the one hand, the storage in Business Critical is faster than remote blob storage queried over http. (That’s not a high bar.)

But on the other hand, if you modify much data, this is a LOT of nodes to sync to. You can run into HADR waits based on syncing to AG nodes, especially if you have a Business Critical partner in a Failover Group, and this can slow your write workload.

Business Critical might be a better fit for you than General Purpose, depending on your workload, your write patterns, your budget, and your requirements for uptime, but I think in most cases you’ll be better off with the General Purpose service tier using memory-optimized instances to compensate for slow storage speeds, and perhaps a Failover Group to meet HA/DR requirements if you need protection in more than one region. (Yep, you pay extra for a failover partner. But if it’s General Purpose as well, 2 GP instances are still significantly cheaper than 1 BC instance with the pricing above.)

Or maybe your best fit is a different service like Amazon RDS for SQL Server, or a VM solution.

Azure SQL Managed Instance General Purpose Architecture and Performance/Cost Tradeoffs

help@KendraLittle.com (Kendra Little) — Mon, 22 Jan 2024 09:26:35 -0500

Whether or not you use Azure SQL Managed Instance, you will likely be asked for an opinion on it eventually if you’re a SQL Server person.

While the architecture is documented, it can be a bit of a long read and some of the gotchas are spread out over different pages – so I’m drawing up the architecture of each service tier along with notable implications for the design on performance and cost. Here’s the scoop on General Purpose.

Architecture for Managed Instance in General Purpose

Here’s the basic architecture for the General Purpose Service Tier. It’s basically a failover cluster style design with one active virtual compute node and shared storage.

Gotchas: Storage

The big old gotcha is that the shared storage is Azure Blob Storage, which is not renowned for its speed. There is also a maximum number of IOPs based on data file size which maxes out at a throughput of 250 MiB/s– your IO will be throttled if it exceeds this.

Because of the pokey storage, the docs for General Purpose notes regarding failovers that:

a heavy workload might experience some performance degradation during the transition since the new database engine process starts with cold cache.

No kiddin'.

There’s an instance limit on maximum log write throughput, too. For transaction log writes on General Purpose you will be cruising at or below 4.5 MiB/s per vCore, Max 120 MiB/s per instance, and 22 - 65 MiB/s per DB (depending on log file size).

What About Memory?

While you can provision memory to compensate/lower physical reads for data files, there’s no individual ‘memory" slider– the amount of memory you have is proportional to your number of vCores:

5.1GB RAM per vCore under Standard Series hardware: 8 vCore = only 40.8GB RAM
7GB RAM per vCore under Premium Series: 8 vCore = only 56GB RAM
13.6GB RAM per vCore under Premium Memory Optimized: 8 vCore= only 108.8 GB RAM

As someone who had multiple clients with 2TB of RAM on failover clusters 8 years ago on on-prem hardware, these numbers still take some getting used to. Busy SQL Servers eat memory like tourists at a Vegas luxury buffet.

In effect, you’ll often need to overspend on CPU if you want to use buffer pool to reduce physical IO for larger databases – make sure to factor that into cost estimates. And you’ll probably also end up having larger data files and log files than you might need, as well, in order to get more IOPs. (Yep, you pay for storage over 32GB, based on how big those files are, too.)

Pricing Examples: General Purpose and Business Critical

With all these tradeoffs, you might think, “Why not just use Business Critical for databases with high performance requirements?” Well, Business Critical has its own gotchas that I’ll dig into in a future post, but arguably the biggest differentiator of the Business Critical Tier is price.

Using the East-US region as an example and assuming the instance needs to be on all the time for one year (8,760 hours):

16 vCore General Purpose x Standard Series = $4.035/hour pay-as-you-go = $35,346.60/year
16 vCore General Purpose x Premium Series Memory Optimized = $5.541/hour pay-as-you-go = $48,539.16/year
16 vCore Business Critical x Standard Series = $10.871 pay-as-you-go = $95,229.96 /year
16 vCore Business Critical x Premium Series Memory Optimized = $13.883/hour pay-as-you-go = $121,615.08/year

For simplicity I only listed Standard Series and Memory Optimized at 16vCores– use the pricing calculator if you want to see other options. These pricing examples don’t include Failover Groups— multiply by 2 to estimate that.

Yep, you can pay $121K USD per year for — let’s see— only 217.6 GB of RAM if you’re at 16 vCores in the Business Critical Service Tier with Premium Series Memory Optimized. (Blinks)

You can lower pricing using reserved capacity, but make sure you read limitations on Self-service exchanges and refunds for Azure Reservations before committing. There are limitations on the refund policy like, “Azure doesn’t process any refund that exceeds the 50,000 USD limit in a 12-month window for a billing profile or EA enrollment.” So be very, very sure before you commit to reservations.

Will Managed Instance Pricing Go Down in Some Way?

I do think Managed Instance is a great idea: I don’t want to administer failover clusters or Availability Groups myself– and it’s rare that I meet anyone who loves doing that. Managed Services are very attractive to alleviate that pain point. And I do think that automated administration and management of these services is worth paying for.

However, this is a VERY expensive service. I hope the value per dollar goes up in the future, both in terms of keeping Managed Instance up to date with features and in giving you more memory and faster storage for your dollar. In my (completely anecdotal personal opinion), price is one of the bigger blockers for adoption, as Aaron Bertrand explains regarding Stack Overflow’s Azure Migration.

Check out more posts in this series: Is the Azure SQL Managed Instance Business Critical Service Tier Worth the Cost?

Automatic Plan Correction Could Be a Great Auto Tuning Feature for SQL Server: Here Is What It Needs

help@KendraLittle.com (Kendra Little) — Wed, 17 Jan 2024 00:00:00 +0000

🔥 UPDATE: The sys.sp_configure_automatic_tuning stored procedure is now documented and supported by Microsoft. Thanks to the SQL Server Product team for this improvement.

I’ve written a bit about SQL Server’s Automatic Plan Correction feature before– I have an hour long free course with demos on Automatic Plan Correction here on the site.

Today I’m updating that course with a note: after using Automatic Plan Correction in anger for a good amount of time, I do not recommend enabling the feature. I’ve had it cause too many performance problems, and there are not a ton of options for an administrator when it’s causing those problems.

Meanwhile, becoming reliant on the feature for the places where it does help makes it difficult to disable the feature. You end up stuck with a very weird set of problems that are oddly similar to the problems the feature was designed to solve.

Further investment in the feature could solve these problems and make this a great tool for customers. Here’s a run down of what Automatic Plan Correction needs from a user who has suffered from it.

It’s an All or Nothing Feature Because the sys.sp_configure_automatic_tuning Procedure was Never Finished, Documented, and Supported

Picture it: you’re a DBA or developer who gets an alert that a key part of your application is running too slowly. You dig in and find that there’s a slow query causing the problem.

You think, “maybe it’s parameter sniffing?” You use a tool like sp_BlitzWho to identify that the query is running long, and it even gives you a handy command to bump the query plan out of the shared plan cache so that a new plan might be compiled. You run that command.

Nothing changes. New sessions running the query get the exact same plan, no matter how many times you bump it out of the cache.

You start digging around and you finally find the plan in Query Store. You see that the plan is forced. If you look into sys.query_store_plan, you see that the plan_forcing_type is ‘Auto’.

This means that the automatic plan forcing feature is enabled, and has forced a plan in Query Store in an attempt to help performance. It may be still testing out if the plan is better or not, or it may actually believe that it’s better – you’d have to go query the sys.dm_db_tuning_recommendations table to figure that out. (Better do that fast, or the info may get cleared out– it doesn’t get persisted across restarts/failovers,etc, and it also might be overwritten after statistics change.)

How do you fix it?

Short term: you can manually un-force the plan in Query Store. This will allow a new plan to be generated. The problem may come back, however, because un-forcing a plan doesn’t exclude a query from future automatic tuning cycles. (When will those restart? I donno, I don’t think it’s documented. But it’s not rare.)

Long term: you can add a recompile hint to the query. This will prevent Automatic Plan Correction from picking it up, but the trade off is that you are going to burn more CPU for that query every time it optimizes. If the query runs very frequently, or if you find a problem where a large set of queries has a regression (like I describe below with temp tables), that becomes a pretty expensive option.

I suspect that the original design of the feature meant for users to be able to address this in different ways. Erik Darling noticed that there is a procedure in SQL Server named sys.sp_configure_automatic_tuning which is not documented, and filed an issue asking about it. This procedure looks like one could use it in this way:

EXECUTE sys.sp_configure_automatic_tuning
 @option = N'FORCE_LAST_GOOD_PLAN', /*Option*/
 @type = N'QUERY', /*Target*/
 @type_value = 1, /*query_id from Query Store*/
 @option_value = N'OFF'; /*Disable*/

Although Erik notes that he saw this worked, I’ve had more mixed results in testing. It sort of worked– but a new query id for the query I excluded was generated in Query Store and then automatic plan forcing went after THAT.

This leads me to believe that the feature to manually configure plan forcing per query wasn’t completely finished, and it was cut from scope in order to get the feature shipped. (I am not a Microsoft MVP. I have no special knowledge here. I’m only guessing based on my own experiences as a product manager at various software companies.)

I have found that the feature is badly needed, however, and I think it’s worth the time/investment for Microsoft to get sys.sp_configure_automatic_tuning completed and working.

Some Queries are Not Suitable for Repeated Experimentation

So, which kinds of queries might you want to exclude from Automatic Plan Correction? I’ve found there are two big groups. One of them could be fixed by Microsoft without fixing sys.sp_configure_automatic_tuning– but this one, not so much.

The first group is queries that can’t afford experimentation. If you run a high performance OLTP system, you are likely going to have some queries that need to always be very fast. The Automatic Plan Correction feature works by experimentation: after you get several hundred runs of a query, if it notices some are slower than others, it will try forcing some of the faster execution plans, then observing if they get slower. If it regresses, it’ll unforce the plan and try another one. (Exactly how many will it try if you have many many plans? I’m donno, I don’t think this is documented. It seems like a lot.)

Even if Automatic Plan Correction finds a plan to force that is nice and fast, it won’t last forever. Statistics updates, failovers, or other situations will cause the experimentation cycle to restart.

This results in uneven performance, and a LOT of confusion whenever code on these mission critical queries is updated, as it can take a while for the plan forcing to kick in. There are two things to help this situation:

Give users the ability to exclude individual queries without recompile hints via a working sys.sp_configure_automatic_tuning procedure.
Give users data that is persisted longer so that they can identify plans that are repeatedly forced, which ones regress, and which ones don’t. These can be either manually forced, or the queries and indexes can be tuned to stabilize performance. Currently, not enough data is persisted over time to make this workable without a huge time investment on the user’s part to regularly persist, groom, and analyze the data. But this ability could make the pain of uneven performance worth it– because it would give information needed to help resolve the issues longer term.

Automatic Plan Correction Invented “Temporary Table Sniffing” Problems – and They Hurt

Another class of queries that you likely want to exclude from Automatic Plan Correction is queries that read from temporary tables.

One of the most common performance tuning techniques used with SQL Server is to break down complex queries into separate, smaller steps, using a temporary table to persist data calculated by the initial query. This helps you not create single massive queries with 50+ joins in them by staging a set of data with the core of your result set, then joining to it to get details in further steps.

Temporary tables are often recommended for this because they support column and index statistics. These statistics help the SQL Server query optimizer understand how much data it is working with, and recompile query plans if the size of the data in the temporary table one one run is significantly larger than the last time it ran.

Automatic Plan Correction breaks this performance tuning technique, in my experience. It will freeze query plans that read from temporary tables and prevent them from recompiling. I’ve seen some truly ridiculously bad performance from this happening.

This is ironic/sad when you start to think about it. Automatic Plan Correction as a feature is designed to help with parameter sniffing problems, where a plan gets cached that may not suit the variations of parameters that are passed in on later executions. But with temporary tables, it basically creates a different temporary table “sniffing” problem: when it freezes plans, it is “sniffing” the plan based on the statistics coming out of the temp table in that iteration of the plan. So it creates a different flavor of the problem it’s designed to solve.

I have code that reproduces this issue with the Stack Overflow database that I’ll publish in a future post– it’s not that hard to show, but this post is big enough.

Two things could solve this

Automatic Plan Correction could automatically exclude queries that read from temporary tables when there is a large number of plans (maybe greater than 10 or 15). This high number of plans likely indicates that the temp table is causing natural recompiles that are actually beneficial for performance.
The sys.sp_configure_automatic_tuning procedure could be finished and given options to handle this type of exclusion.

Is It Helping You Overall? or is It Hurting Your Performance More? the Data Doesn’t Stick Around Long Enough to Find Out.

I think Automatic Plan Correction has a lot of the ingredients to make an impressive feature that does a very important thing: save customers money.

If a feature can make it possible for customers to demonstrate that enabling it saves them money, people will start shouting about it. They’ll recommend it to others. It’ll take off.

That’s not happening now. Currently, Automatic Plan Correction is enabled by default in Azure SQL Database and Azure SQL Managed Instance. Some customers disable it. I almost never hear of customers who don’t use Azure SQL who enable it. Part of that is likely that they never hear about it, because nobody is shouting about it. Part of it is probably hesitancy to turn something on without a clear way to understand and show if it is helping or hurting.

To fix this

Data from sys.dm_db_tuning_recommendations needs to be able to be persisted. Right now, you can only see data for current recommendations. There is only one row per query_id. What experiments were performed last week? Before the last restart? How many experiments have been performed overall for a given query, and what were the outcomes? Make it possible – ideally simple– for users to figure that out, and you’ve given users the tools to understand if you are saving them money. That’s a huge tool. You’ll get champions, adoption, increased advancement in the feature. This will also lead to greater product stickiness for SQL Server. (Could I rig this up myself as a user with duct tape and stubbornness? Probably. Do I have the time and think it’s worth the business investment for this code to be managed by an individual over time? Nah, it should be part of the feature.)
A variant of this functionality could also exist when the feature is not fully enabled, based on the recommendations it makes alone. Reports could estimate how much savings might be realized if the feature is enabled.

Fun Times with Automatically Forced Plans in a Failure State Which Don’t Un-force Themselves

In my experience, it is not uncommon for automatic tuning to force a query plan which ends up in a “failure” state. You can view this in the force_failure_count and last_force_failure_reason columns of sys.query_store_plan. Generally I see the last_force_failure_reason to be GENERAL_FAILURE.

In theory, this shouldn’t necessarily create a big problem. And in my experience, it doesn’t usually cause a big problem. However, once I have had it cause a VERY weird problem of very extended compilation time for a query. The problem went away when the query plan forced that was in the GENERAL_FAILURE state was removed.

In either case, however, what I’m seeing is that Automatic Tuning is forcing a query plan, and then that plan forcing ends up failing– and Automatic Tuning doesn’t notice and clean that up. It should notice and clean that up.

Automatic Plan Correction has Great Potential and is Worth Further Investment

Although I don’t recommend using this feature now, I think the issues with this feature are simply due to lack of continued investment and maturing the feature by Microsoft. That’s very fixable.

I do believe that in many cases this feature can solve a lot of gnarly parameter sniffing problems! I wish I could recommend it. It just needs a few more levers to be added and to make it easier for users to analyze and share when it is working well, and if it is actually saving them resources and time.

Power BI Performance Tuning with Eugene Meidinger

help@KendraLittle.com (Kendra Little) — Tue, 16 Jan 2024 00:00:00 +0000

Eugene Meidinger stops by the Dear SQL DBA Podcast to chat about Power BI Performance Tuning.

We talk about the various engines and languages used in Power BI and big-picture strategies for getting performance from the start. Eugene then talks about the community of tools and techniques that can be used to dig in and solve performance problems in Power BI.

Pick Your Favorite Way to Listen to the Episode

Subscribe on Apple Podcasts – if you leave a five star review, I appreciate you! That helps more folks find the podcast.
Plug this URL into your favorite podcasting app: http://dearsqldba.libsyn.com/rss

Or Watch on Youtube

Youtube.com/@Kendra_Little

Want to Learn More About Performance Tuning Power Bi?

Eugene’s new Pluralsight course on Perf tuning Power BI launches on February 5th. The first 50 sales will be only $20. If you don’t get one of those lucky spots, use code RACOONS to get 50% off.

Find more of Eugene’s content at sqlgene.com.

Find More from the Dear SQL DBA Podcast

This is Episode 80 of the Dear SQL DBA Podcast.

Microsoft is Breaking Database Compatibility Levels for SQL Server

help@KendraLittle.com (Kendra Little) — Mon, 18 Dec 2023 00:00:00 +0000

According to Microsoft’s documentation, ‘Database compatibility level … allow[s] the SQL Server Database Engine to be upgraded while keeping the same functional status for connecting applications by maintaining the same pre-upgrade database compatibility level."

But these days, the “functional status” of a database at a given compatibility level differs depending on whether you’re using SQL Server, Azure SQL Managed Instance, or Azure SQL Database – and in the hosted versions it may change anytime without notice. Surprise, your database behaves differently now!

The whole concept is breaking down, and this is bad news for users of both managed services and the boxed product.

An Example: Database Compatibility Level 150 (SQL Server 2019)

In November 2019, database compatibility level 150 became the default for new databases created in Azure SQL Database.

Compat level 150 introduced intelligent query processing features. However, which query processing features were enabled if you use that compat level changed over time.

The scalar UDF inlining feature was not available in Azure SQL Database or Azure SQL Managed Instance until September 2021, nearly two full years later.

At that point, if you were already at compat level 150, scalar UDF automatic inlining just … started… happening. Maybe you saw the blog post, maybe you didn’t. That “functional status” of the database moved a lot right beneath your feet.

That feature is basically magic, but magic is dangerous. Some queries get faster, some queries get slower.

This has Become Obvious with Database Compat Level 160 (SQL Server 2022)

I wrote in a recent post that SQL Server 2022 Intelligent Query Processing features are hugely incomplete in Azure SQL Managed Instance. (I think this is true for Azure SQL Database as well, but to be honest it’s a lot to keep straight, so let’s use MI as the example.)

None of the following intelligent query processing features are available in Managed Instance:

Cardinality Estimation Feedback
Memory Grant Feedback (percentile)
Memory Grant, Cardinality Estimation, and DOP feedback persistence
Optimized plan forcing
Parameter Sensitivity Plan Optimization

But let’s say you want to enable database compatibility level 160 so that you can use something like the new generate_series function or using enable_ordinal in string_split.

When will all those performance-changing features listed above magically become enabled?

How will you know?

If they cause a regression, how will you be able to tell what changed, and how long will it take you?

Dear reader, I am not excited to find out for myself, either.

This Leaves “Boxed Product” SQL Server Users Feeling Experimented On

The biggest group of users for SQL Server are still those who buy licenses and run the “boxed product” version, whether they are doing so on-prem or in VMs in the cloud. (I don’t say this from any official data, but from chatting with users and doing things like polling a room of ~100 students at a performance tuning day long session at conferences. There is some adoption of managed services for SQL Server, but it appears to be pretty limited, still.)

These users tell me they wonder: why aren’t these features available in the Managed Services?

Will they cause instability in Query Store, which then causes it to not work?
Do they not fully work as designed?
Do they cause a lot of regressions if you enable them?

There is no way to know, because Microsoft hasn’t documented why these features aren’t available in Azure SQL Database or Managed Instance. Among users, I hear a lot of suspicions that if these features were effective and reliable, surely they would be made available to the people who Microsoft is working very hard to sell lucrative Managed Services to.

“There are Database Level Controls,” Some might Say. That Makes It Worse for Managed Instance Users, Not Better.

There are database scoped configuration options for many of these features. Maybe all of them, it’s kind of hard to read the list in the document with everything being in all caps and the names using abbreviations/ different wordings.

It is theoretically possible for a Managed Instance user who doesn’t want features to suddenly appear with minimal warning to:

Diligently go through the documentation and manually disable all the settings for features which are not yet available for the service they are using in each and every database.
Write automation to do this in new databases/instances.
Manually review blogs and documentations every… I don’t know how often… to see if the features are available and then plan to test them.

There may be one user in the world who does that. Maybe two or three. It’s a bad idea, though.

This style of configuration is way too complicated. You’d have to document the reasoning all yourself and keep it up to date. It’s fragile: how likely is the upkeep of this configuration management likely to last if the person leading it changes teams? And it’s not even recommended by Microsoft: all the documentation says to rely on database compatibility levels. Going in this direction will only lead to weirdly configured instances that are hard to troubleshoot in the long term.

This approach is not why people want to adopt a hosted service for databases. It’s pretty much the opposite. And these are the kinds of problems that database compatibility level is supposed to make easier for users.

Make Database Compatibility Levels Make Sense Again

Dear Microsoft Senior leadership, there are two ways to handle this. Don’t let a database compatibility level mean totally different things in products which are supposed to be related, and which you can even fail over to one another.

When you make a database compatibility level available in the SQL Server engine, ensure you make all the same features for that compat level available wherever it works as a standard practice.

Or…

You can ditch the concept of database compatibility levels altogether and stop promising that Azure SQL Managed Instance is always up to date with the latest SQL features and functionality.

But you need to pick one.

Getting Around Error 40510 to Configure Resource Governor in Azure SQL Managed Instance

help@KendraLittle.com (Kendra Little) — Fri, 15 Dec 2023 08:42:12 -0500

One feature I’ve not appreciated enough in the past in SQL Server is Resource Governor.

Resource Governor allows you to fix problems with Memory Grants in a simple way, as Erik Darling recommends. It also lets you classify sessions into groups and limit the maximum number of simultaneous requests, and /or limit the degree of parallelism if you need more CPU for other workloads. While this will slow down the queries you classify into that group, this can be super useful, especially if you’re already using something like Snapshot isolation to prevent blocking or you are using a read-only replica.

In Azure SQL Managed Instance, you get to use Resource Governor, even in the General Purpose tier. This is awesome.

Just make sure you execute commands in the context of the master database, or you’ll get error 40510: Statement 'ALTER RESOURCE GOVERNOR' is not supported in this version of SQL Server.

It is supported! Just USE master; before running your commands.

Directing traffic

At First, I Thought This was a Missing Feature

When I first tested this, I believed the error message. I thought perhaps the documentation was misleading and that the feature was only available in Business Critical, or in some way limited.

Then I realized that I was executing the commands in the context of a user database. You can do this in the on-prem version of SQL Server, but Azure SQL Managed instance is architected a bit differently. When you start looking at your logs and system objects, you’ll notice things like database names actually being uniqueidentifiers, and lots of other little things you don’t see in an on-prem SQL Server.

There are also some handy unique features, like the sys.server_resource_stats view, which persists some performance data for ~14 days, even over restarts.

But basically, some things like this work a little differently in Azure SQL Managed Instance, and I believe it’s related to the cloud-based architecture of the product.

I Changed My Database Context to Master, and It Works Perfectly

Just include a use master; statement at the top of your scripts and you won’t have an issue.

A Note About This is Now in the Docs

I think one of the most common commands folks use for Resource Governor these days is ALTER WORKLOAD GROUP, like Erik suggests in his post. I submitted a PR to the docs to add a note to that page, which has been merged.

Hopefully this makes the ride smoother for others who need to do a little traffic direction for their workloads on Managed Instance.

How Many Features Are Missing from Azure SQL Managed Instance?

help@KendraLittle.com (Kendra Little) — Fri, 08 Dec 2023 21:45:01 -0500

🔥 UPDATE (November 2025): Since this post was written, most Intelligent Query Processing features are now available.

According to Microsoft documentation, most Intelligent Query Processing features are now available in Azure SQL Managed Instance, including Cardinality Estimation Feedback, Memory Grant Feedback (percentile), and Parameter Sensitivity Plan Optimization. Some features require specific database compatibility levels.
Storage increases: Business Critical service tier now supports up to 16 TB of storage (increased from 4 TB), and the new Next-gen General Purpose service tier supports up to 32 TB. See resource limits documentation for details.
Next-gen General Purpose (GPV2) improvements: The new Next-gen General Purpose service tier eliminates the need to configure abnormally large file sizes to get IOPS and throughput. This limitation only applied to the original General Purpose (GPV1) tier, where file sizes determined IOPS and throughput allocation.

Still missing: The following features remain unavailable in Azure SQL Managed Instance as of November 2025, along with most of the features in the post not listed above.

Optimized plan forcing with Query Store
Memory optimized tempdb metadata (SQL Server 2019 feature)
Writable Query Store on Readable Secondaries

Spoiler: a large amount of features from SQL Server 2022 are missing from Azure SQL Managed Instance. Some major features are missing that were introduced in SQL Server 2019– and here we are just a few weeks away from 2024.

But Microsoft’s top-line marketing claims about Azure SQL Managed Instance remain that ‘it’s always up to date with the latest SQL features and functionality.’

Let’s dig into some of the documented highlights on missing features, so you can decide for yourself what to think of that statement.

The Marketing Claim vs Reality

On its main product page, the pitch for Azure SQL Managed Instance is:

Always operate on the latest version of SQL. Stop worrying about updates, upgrades, or end of support. SQL Managed Instance is built on the SQL Server engine, so it’s always up to date with the latest SQL features and functionality.

Managed Instance users DO need to worry about upgrades, because they are constantly confused about what Managed Instance can and can’t do at a given time compared to “normal” / boxed-product SQL Server. Some information is in documentation, but it’s scattered about in many places. Other limitations, well, users discover them on their own.

SQL Server 2022 Features: Missing or Incomplete

Brent Ozar has written about how Azure SQL Managed Instance Link, a big hero of the SQL Server 2022 release, was in a gated private preview that could only be accessed after filling out a form and waiting in a queue of indeterminate length. The feature is now in Public Preview (ungated) as of November 15, 2023, but as Brent writes: “It’s amazing to me that even a year after its release, SQL Server 2022’s flagship feature still isn’t ready yet.”

Writable Query Store on Readable Secondaries remains in preview for the boxed product, but this feature is not available at all in Azure SQL Managed Instance. I see no way to even preview it.

Most Intelligent Query Processing features are now available in Azure SQL Managed Instance (see the update note at the top of this post). However, Optimized plan forcing with Query Store remains unavailable as of November 2025.

SQL Server 2019 Features Still Missing

I wrote a feedback article on the lack of the SQL Server 2019 feature memory optimized tempdb metadata in Managed Instance. Essentially, tempdb metadata contention occurs on system tables when you use tempdb at higher volumes. Unlike other types of tempdb contention, this can’t be alleviated by adding multiple data files, as these system tables only ever reside in one data file in tempdb. But if you hit this problem in Managed Instance, the feature that fixes it isn’t available to you.

Only recently, Microsoft made it possible to add data files to tempdb in Managed Instance. However, you might notice that there are commands to add files and remove files, but NOT to set the file size or to grow files. How do you make tempdb files the same size in Azure SQL Managed Instance to follow Microsoft’s own performance guidelines for tempdb? I don’t see how one can.

Database and Instance Limitations

You can’t have more than 100 databases per instance. Not all databases are of equal size, activity, or usage, so this cap seems weirdly arbitrary to me. But there it is.

You can’t get minimal logging (no Simple or Bulk Recovery Models). Minimal logging is really nice for staging data and temporary workspaces. For many of these use cases, data loss of staged data is not a huge problem– you’d start the process over again anyway. However, you can’t do this anywhere but tempdb on SQL Server Managed Instance. Remember those limitations for tempdb I mentioned above? Not fantastic.

You can’t lower the max server memory. Would you like to test how things perform if you scale down to have less memory? Well, the normal way to do this won’t work, time to get on your bad idea jeans.

You can’t take database snapshots or use DBCC CLONEDATABASE. DBCC CLONEDATABASE is really handy and useful. Sigh.

You can’t change the time zone after the instance is created. Hope you don’t want to.

You can’t use single user mode. This one I kind of understand, because they don’t want their processes to be locked out– but it still means that you need to create workarounds if you use this for maintenance.

Maintenance and Troubleshooting Gaps

You can’t run DBCC CHECKDB with options REPAIR_ALLOW_DATA_LOSS, REPAIR_FAST, and REPAIR_REBUILD. Hopefully you don’t need to, but it’s good to understand this ahead of time.

You can’t change the number of error logs, or even keep your error logs. This one makes me sad:

“Error logs that are available in SQL Managed Instance aren’t persisted, and their size isn’t included in the maximum storage limit. Error logs might be automatically erased if failover occurs. There might be gaps in the error log history because SQL Managed Instance was moved several times on several virtual machines.” Docs

You can’t use instant file initialization, so you may wait for file growths. Azure storage ain’t famous for being fast.

“A SQL Database managed instance does not use instant file initialization, so you might see additional PREEMPTIVE_OS_WRITEFILEGATHER wait statistics since the date files are filled with zero bytes during file growth.” docs

Features That Just Aren’t There

You can’t use Filestream, Filetable, Fulltext Semantic Search, Merge Replication, Peer to Peer Replication, or Replication with Updatable Subscriptions. I thought a lot of these features were overhyped/not great anyway, but a lot of people disagree with me about that.

You can’t use some columnstore index features– maybe? Hard for me to tell if this doc applies to managed instance or if it’s up to date.

You can’t use event notifications. I actually liked that feature. Maybe I was the only one.

Log Shipping and Backup Limitations

You can’t use log shipping for anything other than migrating to MI. The only log shipping service available in Managed Instance is the Log Replay Service. As far as I can tell from reading, that only works with Managed Instance as the destination. If this is incorrect, please correct me in the comments.

Isn’t it always a little creepy when it’s easier to get into something than it is to get out?

Also note:

LRS is the only method to restore differential backups on managed instances. It isn’t possible to manually restore differential backups on managed instances or to manually set the NORECOVERY mode by using T-SQL.

Storage and Performance Quirks

You can’t have reasonably sized data and log files in General Purpose. “In the General Purpose service tier, every database file gets dedicated IOPS and throughput that depend on the file size.” Docs Having to create artificially large files gets real weird, real fast. And of course you get to pay for the storage.

I Know There’s More

I know there’s more, and I’ve probably missed something obvious. But you get the picture.

How do We Fix This?

I don’t think it’s our responsibility as customers to fix this. I’ve been told by Microsoft folks that I should create “suggestions” for missing features in their forums, but… really?

Why should users need to file suggestions that a vendor follow the primary pitch for their own product? My time and your time is valuable, friends. We shouldn’t have to write suggestions and vote so that someone can pick and choose how they live up to their own commitments: they already committed to doing that.

If you are considering or using Managed Instance, perhaps let Microsoft Sales know that you are concerned about the number of missing features in Azure SQL Managed Instance, and that it’s important to you to have the latest features from SQL Server available. Not the imaginary ones, the actual ones.

Lost Updates Under Read Committed Snapshot Isolation (RCSI)

help@KendraLittle.com (Kendra Little) — Wed, 04 Oct 2023 08:28:49 -0400

I shared an image on social media this week that describes how I feel about isolation levels in SQL Server (and its various flavors): the more concurrent sessions you have in a database reading and writing data at the same time, the more attractive it is to use version-based optimistic locking for scalability reasons.

There are two isolation levels in SQL Server that use optimistic locking for disk-based tables:

Read Committed Snapshot Isolation (RCSI), which changes the implementation of the default Read Committed Isolation level and enables statement-based consistency.
Snapshot Isolation, which provides high consistency for transactions (which often contain multiple statements). Snapshot Isolation also provides support for identifying update conflicts.

Many folks get pretty nervous about RCSI when they learn that certain timing effects can happen with data modifications that don’t happen under Read Committed. The irony is that RCSI does solve many OTHER timing risks in Read Committed, and overall is more consistent, so sticking with the pessimistic implementation of Read Committed is not a great solution, either.

Race conditions / timing issues under RCSI

Like many folks, I chat with various AI assistants these days. Although their claims require fact-checking, they help me understand concepts I’ve had a hard time getting my head around, as well as improve my wording on concepts I already DO understand. And unlike some forums, AI doesn’t chastise me that my question’s already been asked by user GoblinFeet1000 and will reword things without judgement until I catch on. (Aside: one example is the Halloween problem.)

So I gave GitHub CoPilot the prompt: “what specific kinds of lost updates can happen under read committed snapshot isolation level in sql server”

Here’s my improved summary of RCSI’s timing issues after its help:

Under RCSI, readers generally don’t block writers, and writers don’t block readers. This means it allows transactions to read data that has been modified by other transactions but not yet committed. If one transaction reads data that another transaction has modified but not yet committed, and then attempts to update that data, the second transaction’s changes may be lost when the first transaction commits. This doesn’t happen with every type of modification, but some query plans for modification allow this risk.

But even re-worded, that explanation can make people pretty nervous to enable RCSI. (Note: if you’d like a technical post on timing issues under RCSI, Paul White has written a great one.)

RCSI fixes many timing issues you already have with Read Committed

Read Committed has plenty of its own timing issues, it’s just that most people figure they already have it and nobody is complaining, so it’s “comfortable” as the status quo.

Under Read Committed:

Queries can miss rows entirely if they are modified in a way that relocates them in an index while a scan is ongoing.
Similarly, rows can be counted twice for the same reason.
Non-repeatable reads are allowed in transactions, so if you query something twice, it may have changed without explanation. (This can even happen within the a single query plan if it needs to read data twice.)
Phantom reads are allowed in transactions, so if you read something twice something may have been added without explanation.

If good accuracy is needed for data in a highly concurrent situation, it’s not a great solution to stick with this!

According to ChatGPT, that would be like:

Refusing a medicine that cures six ailments because it might cause one side effect.
Not using a safety net while tightrope walking because it might snap, even though it would save you from multiple deadly falls.
Not to carry an umbrella because it might get windy, even when the forecast predicts heavy rain.
A traveler in the desert avoiding an oasis because of a rumored snake, while ignoring the clear water that could save them from thirst.

The way forward: protect your modifications

One essential thing is to recognize that protecting yourself against data inaccuracies with timing conditions is a requirement for using databases, no matter what isolation level you use.

For your data modifications, you can protect them from the risk of edge conditions by examining them and identifying what isolation level they actually need. Your queries that do modifications perhaps should run under repeatable read, serializable, or snapshot isolation level to avoid race conditions, depending on your patterns and your requirements. This can co-exist with RCSI– it doesn’t replace the need to sometimes use a different isolation level, it only changes how Read Committed is implemented.

If both scalability and data correctness matter, this is worth your time (even if you stick with pessimistic Read Committed as your default). Start by identifying the most frequent and most critical transactions that do modifications for your application, and identify what level of protection they need.

And if you decide that the risk of a few edge conditions is tolerable after looking at this, you might just enable RCSI like Michael J Swart’s team eventually decided to do.

Automated Deployments and the Art of the Database with Database Michael J Swart

help@KendraLittle.com (Kendra Little) — Tue, 26 Sep 2023 11:39:38 -0400

In this episode, Michael J Swart joins the Dear SQL DBA Podcast to talk about databases, automation, and how he’s come to illustrate some of the coolest blog posts ever to be written about data.

About Michael J Swart

Check out Michael’s art and blog posts at michaeljswart.com, and explore his posts by illustration at https://michaeljswart.com/browse-articles-by-illustration/.

Find more from the Dear SQL DBA Podcast

This is Episode 79 of the Dear SQL DBA Podcast.

Pick your favorite way to get more free content:

Youtube.com/@Kendra_Little
Subscribe on Apple Podcasts
Plug this URL into your favorite podcasting app: http://dearsqldba.libsyn.com/rss

PAGELATCH, PAGEIOLATCH, and LATCH Waits in SQL Server

help@KendraLittle.com (Kendra Little) — Tue, 12 Sep 2023 09:59:09 -0400

I’ve long found it tricky to remember and explain the differences between three similar-sounding waits in SQL Server that all have “LATCH” in the name: PAGELATCH, LATCH, and PAGEIOLATCH waits.

Here’s an illustration that explains these waits, along with wait subtypes.

Quick Comparison

Here’s a table that compares the three latch wait types:

Wait Type	Contains “IO”?	Contains “PAGE”?	Page Location	What It Means
PAGEIOLATCH	Yes	Yes	On disk (storage)	Waiting to read pages from disk into memory buffers
PAGELATCH	No	Yes	In memory	Waiting to access pages that are already in memory
LATCH	No	No	N/A (not a page)	Waiting to access a structure or resource in memory that is not a data page

PAGEIOLATCH Waits

PAGEIOLATCH at least has a giveaway in its name: the “IO” characters indicate I/O, aka “Input/Output” - a reference to transferring data, in this case from storage to memory.

My definition: “I need to get PAGES from DISK (storage) into BUFFERS in memory. To investigate or reduce this wait, find queries with highest reads and look for indexing and tuning improvements.”

Microsoft docs definition: “Occurs when a task is waiting on a latch for a buffer that is in an I/O request.”

PAGELATCH Waits

There’s no “IO” in this one. That’s because the page is already in memory.

My definition: I need to get access to a page that is already in memory. Sometimes this is “last page” insert contention, Page Free Space (PFS) contention, or tempdb contention.

Microsoft docs definition: “Occurs when a task is waiting on a latch for a buffer that isn’t in an I/O request.”

LATCH Waits

There’s no “PAGE” or “IO” in this one. This one is the hardest to explain, too.

My definition: I need to access a structure or resource in memory that is NOT a data page. Can be anything from transaction log management to supporting parallel scans.

Microsoft docs definition: “Occurs when waiting for an [latch subtype] latch. This doesn’t include buffer latches or transaction mark latches.”

Wait Subtypes: EX, UP, SH

First, there are more subtypes than these three (KP, DT) - but these are the most common subtypes I see. Here is how I remember these:

_UP = Update: I need to make some mods but others can read
_EX = Exclusive: I need this to myself, everyone else wait
_SH = Shared: I need to read this

Learn More About Waits

There are tons of great free resources to learn about wait types on the web. Some of my favorites are from Microsoft Learn and Paul Randal of SQL Skills:

List of waits: sys.dm_os_wait_stats (Microsoft Learn)
Transaction Locking and Row Versioning Guide (Microsoft Learn)
Wait Stats Library (SQLskills)
Latch Contention Whitepaper (Microsoft Learn)
Latch Classes Library (SQLskills)
Most common latch classes (SQLskills)

Jer and Kendar Explore Optimized Locking

help@KendraLittle.com (Kendra Little) — Mon, 04 Sep 2023 00:00:00 +0000

🔥 UPDATE (November 2025): Microsoft has introduced optimized locking v2 with significant improvements. The new version includes Skip Index Locks (SIL) and Query Plan LAQ Feedback Persistence, which further reduce lock overhead. The improvements are most pronounced for nonclustered indexes. Optimized locking v2 is available in SQL Server 2025 and Azure SQL.

SQL Server has a new feature that’s currently only available in Azure SQL Database: Optimized Locking.

Jeremiah Peschka joins Kendra (aka Kendar) to talk through the docs and nerd out on locks, blocks, and how to pronounce the acronym “LAQ”.

Prefer to explore optimized locking with a diagram? I’ve also got a little sketchnote for ya.

Podcast episode

Sketchnote of optimized locking concepts

Here’s a sketchnote style comic I drew up of what I’ve learned about the feature so far.

Right click and open in new tab for larger image

Learn more about optimized locking

Learn more about this feature online:

Optimized locking article
Transaction locking and row versioning guide <– A much longer read

Find more from the Dear SQL DBA Podcast and Jeremiah Peschka

This is Episode 78 of the Dear SQL DBA Podcast.

Read more from Jeremiah Peschka at facility9.com.

Dear SQL DBA - Advice for Technical Leaders with Alex Robson

help@KendraLittle.com (Kendra Little) — Thu, 31 Aug 2023 00:00:00 +0000

Ever wondered what it’s like to be a VP or Director of Engineering? I chatted with Alex Robson about leadership in technology, what you can get out of coaching or an MBA program (should you be interested), and what makes a high performing team.

Alex’s site and blog: https://robsonconsulting.services

My chat with Alex Robson

Alex’s content recommendations

Alex shared these recommendations:

I believe Camille Fournier and Will Larson are wonderful writers with invaluable insights and advice.

For product thinking, I recommend folks read The Lean Startup by Eric Ries, Principles of Lean Product Development Flow by Don Reinertsen, Safer Sooner Happier by Jonathan Smart, and Accelerate by Dr. Nicole Forsgren.

Be sure to read books on leadership that are outside of engineering. Dan Pink’s Drive and Eliyahu Goldratt’s The Goal are two of my usual recommendations.

Last but not least - read books that are about human behavior. Both economists and psychologists ask important questions that may help you unlock better ways to relate to and understand others. I love Daniel Kahneman’s Thinking Fast and Slow, and highly recommend Mistakes Were Made (but not by me) by Carol Tavins and Elliot Aronson.

More Dear SQL DBA

Check out the episode index.

Msg 195: STRING_SPLIT Is Not a Recognized Built-In Function Name

help@KendraLittle.com (Kendra Little) — Fri, 25 Aug 2023 00:00:00 +0000

Yesterday, I was writing some Transact SQL to dust off the cobwebs. I got confused when I was playing around with the STRING_SPLIT function, and kept getting the error:

Msg 195, Level 15, State 10, Line 2
'STRING_SPLIT' is not a recognized built-in function name.

I checked my database compatibility level, which must be 130 or higher for the function to be available. My database compatibility level was 160, so that was fine.

The issue? STRING_SPLIT is a table valued function. If your syntax uses it as a scalar function, you will still see a message that it doesn’t exist – but really the issue is that the function exists, but needs to be called with a different syntax.

Let’s look at some examples.

Check and set your database compatibility level

This query will return the compatibility level for the current database:

SELECT
 d.compatibility_level
FROM sys.databases AS d
WHERE d.name = DB_NAME();

If your database compatibility level is less than 130, the STRING_SPLIT() built in function won’t be available.

You can read documentation on the differences between compat levels, which is helpful. It’s usually a good idea to make any compatibility levels in development environments first and burn them in for a while before making a change in production.

To update database compatibility level, run a command like:

ALTER DATABASE
 database_name
SET COMPATIBILITY_LEVEL = 160;

How to NOT use STRING_SPLIT

Here’s what I tried to use that didn’t work:

SELECT TOP (100)
 p.Id,
 split_test = STRING_SPLIT(p.Tags, '<', 1)
FROM dbo.Posts AS p;

and

SELECT
 split_test = STRING_SPLIT('a,b,c', ',', 1);

Both of these return the error:

Msg 195, Level 15, State 10, Line 1
'STRING_SPLIT' is not a recognized built-in function name.

What both of these queries have in common is that I’m trying to use the STRING_SPLIT function in the SELECT clause, as if it’s a scalar function. However, it’s a table valued function, so we need to do this another way.

Examples of successfully using STRING_SPLIT

You can use STRING_SPLIT with values that are already in a table, or with a list of supplied values.

STRING_SPLIT on a column with APPLY

For my first query, I want to run the STRING_SPLIT function against the Tags column in the Posts table.

Since STRING_SPLIT is a table valued function, I can do that with CROSS APPLY or OUTER APPLY syntax, like this:

SELECT TOP (100)
 p.Id,
 split_test.value
FROM dbo.Posts AS p
CROSS APPLY STRING_SPLIT(p.Tags, '<') AS split_test;

Notice that I didn’t name the column value. As a table valued function, STRING_SPLIT has already pre-defined the names of the columns it returns. The ‘Value’ column returns substrings after splitting using the specified separator.

🤓 Aside: This example query uses the StackOverflow2010 dataset. The tag column in post uses a more complex delimiter than a single character, but the STRING_SPLIT function only allows a single character to be specified. This presents some extra data cleansing issues if you want to use STRING_SPLIT for this purpose with that data.

If you are running SQL Server 2022 or higher, Azure SQL Database, or Azure SQL Managed Instance, you can optionally set the enable_ordinal parameter to true to return the “ordinal” column, which indicates the order the substring had in the original string.

The syntax for that is:

SELECT TOP (100)
 p.Id,
 split_test.value,
 split_test.ordinal
FROM dbo.Posts AS p
CROSS APPLY STRING_SPLIT(p.Tags, '<', 1) AS split_test;

STRING_SPLIT on literal values

The following syntax will return a single column named value with three rows, one row containing each letter in the string:

SELECT
 split_test.value
FROM STRING_SPLIT('a,b,c', ',') AS split_test;

I hope these examples save someone a few minutes of confusion.

Dear SQL DBA - Performance Tuning with Erik Darling

help@KendraLittle.com (Kendra Little) — Fri, 18 Aug 2023 16:37:14 -0400

SQL Server performance tuning expert Erik Darling joins the podcast today to chat about how good queries can go bad and how bad queries can get better.

He also answers the question on everyone’s mind: if he was a database, what database would he be?

My Chat With Erik Darling

Good Stuff From Erik

Blog: https://erikdarlingdata.com/blog/
YouTube: https://www.youtube.com/@ErikDarlingData

More Dear SQL DBA

Check out the episode index.

What To Do When They Are Sure It Was The Database... All The Time

help@KendraLittle.com (Kendra Little) — Tue, 15 Aug 2023 00:00:00 +0000

It can be tiring to have the database constantly be the Prime Suspect for performance problems.

I’ve had many variations of this conversation over the years:

“Something happened, and we’re pretty sure it was the database. Again.”

Frequent Vibe Checks Indicate an Underlying Problem

It can be annoying to be interrupted. Sometimes it feels like everyone is always blaming the database. And sometimes it feels like people simply don’t want to do things themselves.

But usually, there’s an underlying problem where people feel like they can’t tell the health of the database without asking a specialist, and probably having to wait for the answer. People don’t usually like to wait, they want to get on with their day.

Does someone asking, “hey, was something up with the database?” have a way to:

Review utilization and query runtime for the time in question?
Check what wait stats were in the time period and compare them to a baseline?
Check the database logs for errors or odd messages?

If they have a way to do this, is it easy for them to remember how to do it and access it? Could a custom report or a set of instructions help them run through the process more easily?

Talk to your questioner about what types of tooling, information, or processes could help them get this information themselves, without having to wait.

I Don’t Have Time for That

These tools or processes may not be things that you can stop and create immediately, and that’s fine.

When having this discussion, think about yourself as a type of “product manager” for working with the database. You aren’t volunteering to do things, and it’s fine to be clear about that. The important thing is to explore and understand your customer’s problem, and why it isn’t easy for them to do this yourself. That can be valuable information on how your team can work with others better, and can be a cause that you can champion (even if you aren’t doing the work yourself) in the future.

It’s also valuable to set expectations clearly. It’s great to tell your coworker that your plate is full, and you may not be able to solve their bigger problem immediately, but that you’d like to learn more about it so that you can share it with your team.

But I’m Just a DBA

More and more, seniority in tech roles isn’t purely about coding skills or deep technical knowledge. It’s also about:

Understanding your customers, their pain points, and why they choose specific tools and processes.
Advocating for ways to help your customers work better, and including them in relevant conversations so they can advocate for themselves.
“Leading up” and get time allocated to get work done for your customers– even if you aren’t doing it yourself.

In other words, situations like this are truly opportunities.

Don’t get trapped in thinking that your value is in being the one to manually answer these questions as they come up on an individual basis. There’s an even greater value you can give in helping equip people to answer these questions themselves! Don’t worry– there will always be harder questions that they will come back to you for later.

No matter how hard you try to automate yourself out of a job, in my experience it only helps your career.

Dear SQL DBA: Tech Interviews in 2023 With Jeremiah Peschka

help@KendraLittle.com (Kendra Little) — Fri, 11 Aug 2023 00:00:00 +0000

Tech interviews are weird and wacky. How did they get this way, and how SHOULD they be? Jeremiah Peschka joins us to discuss.

Thanks to Jeremiah for being the first guest on my podcast! It only took 75 episodes before I had a guest.

More From Jeremiah Peschka

Jeremiah’s blog is facility9.com.

More Dear SQL DBA

Check out the episode index.

Database DevOps Core Concepts: Migration Scripts

help@KendraLittle.com (Kendra Little) — Wed, 09 Aug 2023 10:52:03 -0400

Database migration scripts are a popular and effective way to check database code into version control.

In this post I describe the most common features of popular migration script runners for database code deployment, along with the top gotchas that folks hit when versioning their database code with migrations.

Benefits of Migration Scripts for Database DevOps

Migration scripts enable checking database code into version control in a human readable format. This provides a centralized, standard way to record all changes.

Migration scripts are also useful to:

Implement Continuous Integration processes for database code / automate validation of changes to the code.
Automate change deployment in both development and production environments.

Together, these benefits bring database code into line with processes and standards for application code.

Common Tools Used With a Migrations Approach

Database migration scripts for change deployment are used by tools like Flyway and Liquibase. Some companies write custom implementations of migration script runners.

If you are evaluating building or buying one of these toolsets, this post will help you understand essential functionality and which workflows to tackle first.

Essential Functionality for Database Migration Scripts

Core functionality to consider when adopting or writing a database migration script runner are versioned migration scripts that can be run in a defined order, a way to track migration script status against a specific target database to run scripts only when needed, baseline scripts, and re-runnable migrations.

What Are Database “Migrations” or “Migration Scripts”?

A database migration script changes the state of your database, most commonly by adding, modifying, or deleting objects in the database.

Migration scripts can also add or modify data in objects in the database. An example:

We start with an empty database.
Our first migration script creates a table for bananas and inserts rows.

CREATE TABLE
 dbo.bananas
(
 banana_id integer,
 banana_name nvarchar(50) UNIQUE,
 banana_color nvarchar(20),
 CONSTRAINT PK_bananas
 PRIMARY KEY CLUSTERED (banana_id)
);

INSERT INTO
 dbo.bananas
(
 banana_id,
 banana_name,
 banana_color
)
VALUES
 (1, N'Cavendish', N'Yellow'),
 (2, N'Plantain', N'Green'),
 (3, N'Red Dacca', N'Red'),
 (4, N'Manzano', N'Yellow'),
 (5, N'Burro', N'Yellow');

ℹ️ Info: I didn't include any error handling in this code. Some tooling helps with that automatically, some does not.

The main reason I show this code is: it’s just that simple! Migration scripts can be written in a variety of languages, depending on what the migration script runner supports, but generally these are scripts written in plain SQL that are easy for humans to review.

Migration Scripts Are Numbered and Run in Order

Our second migration creates a table for banana ratings. This table uses banana_id values that are defined in dbo.bananas.

CREATE TABLE
 dbo.banana_ratings
(
 rating_id integer,
 banana_id integer,
 rating_value integer,
 rating_date datetime,
 CONSTRAINT FK_BananaRating_Banana
 FOREIGN KEY (banana_id)
 REFERENCES dbo.bananas (banana_id)
);

The second migration script has a foreign key dependency on the first migration. They need to always run in a specific order.

Migration runners commonly provide a way to number scripts and execute them in the desired order.

Migration Script Status

The log tracking which migration scripts have been run with their status is often kept in a table in the target database.

The migration script runner generally reads from this table prior to a deployment, and updates the table as the deployment progresses.

Baseline Script

What if you need to start with a database with existing objects?

A “baseline” migration script contains all of the definitions of objects in the database, ordered so that dependencies can be met. This is generally contained in a single migration script with a special status to differentiate it from “normal” migrations.

Migration script runners usually only deploy the baseline script to an empty target database. This allows validation of database code for continuous integration.

Re-Runnable Migrations

Sometimes you want a migration to run every time you deploy changes. This might ensure the target database is configured for the appropriate environment (dev, prod, etc), create objects like stored procedures, or ensure that an object or data is in a specific state.

Migration runners sometimes allow specific naming and ordering conventions for re-runnable migration scripts.

What Gets Tricky?

There’s a lot that can get complex when you get into the weeds. The top places where folks get confused are related to rollbacks, shared database environments, and dealing with changes in deployment order/numbering challenges.

Rollbacks

Some tools have options for Rollback scripts, but managing them is complex. In production, you usually want every migration recorded as a “roll forward” because it’s a real deployment, and you want your future builds to play out the same sequence.

Also, everyone hates writing rollbacks manually. Everyone.

Shared Database Environments

Mistakes or necessary changes to scripts are often identified in shared testing or pre-production environments.

If there is no automated, fast way to reset the database (including all its data), then it can be time consuming to validate that the target environment is in the desired state once code is rolled back and scripts are modified.

Numbering While Integrating Changes From Multiple Teams

Code is often not deployed to prod in the same order it is completed.

Let’s say Ravi writes migration script 42, which needs a lot of testing. Sandy releases an important hotfix in migration 43. Now Ravi’s script is out of order. Should they renumber it? What does that mean for all environments where it has already been deployed? What does this mean for future “from scratch” builds?

Enhancing a Migrations Approach With State Comparisons and Auto-Generated Code

Database “state” capture and comparison tools can augment migration script approaches with cool functionality. Vendors often provide this as part of their value proposition.

Those writing rollback scripts will want to use generative AI to help draft and validate changes. Both humans and AI can write bad SQL, so be wise how you use both.

The Value of Getting Database Code Into Version Control Is Worth the Work

Thinking about the gotchas can be daunting. There is a lot to gain even if some portions of your process remain manual, or if you implement automation in stages. A recap of the benefits of database migrations is worth it at this point.

These will give you the ability for many good things:

Clear audit trail of development and deployment code
Lower likelihood of losing work or accidental deployments
Continuous Integration
Less or very limited manual work in deployments

Dear SQL DBA - Data Roles: DBA, DBRE, and Data Engineer

help@KendraLittle.com (Kendra Little) — Fri, 04 Aug 2023 00:00:00 +0000

There are lots of jobs for data folks. In this episode, I discuss three hot job titles: Database Administrator (DBA), Database Reliability Engineer (DBRE), and Data Engineer (DE).

As Explained by Trash

A Bit More Professionally

More Dear SQL DBA

Check out the episode index.

Why Not Call Database DevOps Simply DevOps?

help@KendraLittle.com (Kendra Little) — Fri, 28 Jul 2023 00:00:00 +0000

Sometimes people are annoyed by the term ‘Database DevOps’. Why not call it simply ‘DevOps’? After all, Database DevOps follows the same core principles.

The answer is simple: implementing DevOps is tricky by itself, but most teams are set up to fail when it comes to implementing DevOps for databases. This makes it worth defining as a specialization.

What is DevOps?

Donovan Brown defines DevOps as “the union of people, process, and products to enable continuous delivery of value to our end users.”

Donovan’s definition encapsulates the essence of what DevOps aims to achieve: an emphasis that people are essential, a focus on collaborative processes, and the importance of tooling and automation with the goal to continuously deliver value all the way to the customers who use it.

As Donovan writes:

It is very important to realize that DevOps is not a product. You cannot buy DevOps and install it. DevOps is not just automation or infrastructure as code. DevOps is people following a process enabled by products to deliver value to our end users.

Why is Database DevOps Uniquely Difficult?

I generally see three major challenges with teams implementing DevOps for Database code:

Data is stateful. Blue/Green (sometimes called Red/Black) deployment models don’t generally work for database code, as data may be lost by returning to the state of the data before deployment began. This adds significant complexity to finding the right products and building the right processes.
Key stakeholders get quite nervous. Data loss or data breaches are incredibly expensive and damaging for brands. This has led to much more rigid change management practices when it comes to deploying changes to databases, especially in more regulated industries.
DBAs and Developers are historically misaligned. Database Administrators (DBAs) are often not deeply familiar with tooling and practices that make implementing devops safer. DBAs are often trained to act as the “protectors” of production environments, as well, and to look for everything that might possibly go wrong. This positioning of the DBA role creates significant friction between DBAs and development and DevOps teams, and makes it much harder for these teams to co-create new processes.

Not everyone has all three of these challenges. Sometimes significant cultural groundwork has been laid and DBA and Development teams are aligned on a strategy. This is somewhat rare, however.

I more often see teams who are successful with Database DevOps because there is only a development team or only a DBA team involved: this tends to make things simpler because of the lack of friction that cross-team collaboration can create. However, in these cases there is often a lack of specialized knowledge that would enrich the process because a key team is missing.

But still, the first two challenges of stateful data and wary stakeholders are nearly ever-present and take significant work to manage.

Avoid the Pitfalls of Database DevOps

While these guidelines can apply to any kind of DevOps initiative, they are especially critical when it comes to databases.

1. Spend a lot of time carefully scoping your MVP

The biggest mistake that I see folks make is trying to ensure they can solve the very hardest problems with stateful data at the very beginning. This often takes the form of, “we need to make sure we can fully automate rollbacks for every problem in 30 seconds in production.” Because of the nature of stateful data, different business requirements, and implications of different data architectures, there is no One-Size Fits All approach to this. Even if you solve this for one database, that doesn’t mean you can cut and paste the same approach to others in your organization. Teams can find this hugely disheartening.

The truth is, there is huge value to simply getting your database code into version control. Automating deployments to dev environments and finding ways to reset those environments automatically also will raise the quality of your code and empower developers to work faster. There are a variety of ways to scope your MVP and plan implementations, and it’s important to carefully identify the right one for your teams.

2. Talk About Guardrails Early and Often to Stakeholders

One pitfall I see is folks focussing too much on avoiding failures as the major benefit of DevOps for databases.

Things are going to break. DevOps can help you identify failures earlier in your process and mitigate risk, but implementing the right processes and the right tooling to do this, while getting folks on board to collaboratively build those processes is an iterative effort. There will be errors, there will be failures.

A strategy that helps with this is the concept of guardrails. If you don’t like that term, you can also talk about safety. The important thing to note is that guardrails– aka safety checks– can be implemented in a variety of ways. Automation is great, but takes time to build. Start with human-powered checks first. Make these human checks stronger, more resilient, and use the information you learn to prioritize how to automate these iteratively.

This will take time. It’s important to plan this time into your effort from the beginning, and ensure that leadership is aware that regular investment in the process over time will be required.

3. Culture

A lot of the hardest problems with Database DevOps are not technical: they are human problems. Database administrators in particular can be highly skeptical of change. I know many DBAs and DBREs (Database Reliability Engineers) who rigidly cling to the identity of the Protector of the Data.

As these data folks see it, developers who aren’t experts on the database they are using are just going to mess things up – but it’s the DBA or DBRE who is going to get blamed and have to clean up the mess.

But you know what? This point of view didn’t come from nowhere. This perspective has been set in IT culture for a long time and is still continually reinforced by tech leadership in subtle ways. There is some truth in this perspective.

For many of these teams, many conversations and a lot of listening to concerns are needed to open up possibilities for change. Bridging the gaps between other teams may take significant work as well. Teams need to be brought together in a way that they can effectively teach and learn from one another.

Database DevOps Is Worth the Investment

The payoff goes back to Donovan Brown's definition of DevOps : continuous delivery of value to your end users.

Delivering value to your customers continually can be the difference between success and failure of products and entire companies. Don’t let your databases be the gating factor in your organization’s success.

The Difference Between DBAs, DBREs, and Data Engineers

help@KendraLittle.com (Kendra Little) — Wed, 26 Jul 2023 00:00:00 +0000

Twenty years ago, database administrators (DBAs) were the primary career path when it came to specializing in data management.

Much has changed: development patterns transformed from Waterfall to Agile, DevOps drives automation and shared ownership of code, and cloud services have made many more kinds of PaaS databases, data lakes, and data lakehouses available to organizations of all sizes.

These changes have introduced new and varied career paths for data folks which have different emphases on skill sets. In this post, I talk through the commonalities and differences between DBAs, Database Reliability Engineers (DBREs), and Data Engineers (DEs). Whether you’re a hiring manager or data professional, it’s worth knowing about these roles.

Why does it matter?

We all know that job titles can be meaningless – I once was a DBA with the job title of “Software Development Engineer, Testing,” because of an HR categorization issue. (Did I develop software? No. Did I write tests? Also no.)

However, I think these concepts are useful to think about for a few reasons:

Defining yourself and planning your career using a common language can be useful in broadening your horizons.
When writing job descriptions, these terms are useful keywords to get the job in front of the best candidates.
For job candidates, understanding the breadth of these disciplines is useful when reviewing opportunities. It’s a red flag when any position requires an individual who covers all of the roles that I describe here with significant breadth. While it’s certainly possible to dip your toe into multiple areas, covering all three of these specialties deeply is unmanageable.

The trash can explanation

I asked folks on social media what they think about these three roles. One of my favorite responses came from Erik Darling:

Maybe it’s just that my first DBA job came with the team motto of ‘Data Janitors’, but this resonated with me. It can all be described by trash.

The business explanation

Although trash cans are easy to relate to, they don’t necessarily help you much in a job interview. Here’s a more specific version of how I’ve seen these roles compare in job postings, and in organizations I’ve worked with as an FTE, external speaker, or consultant.

This image is too complex for alt text, but the content is explained in the paragraphs below the image.

Commonalities between DBAs, DBREs, and DEs

There are a few core specialties between all three of these roles, although they are used in different contexts. All of these roles are concerned with how data is:

Stored
Secured
Shared
Backed up
Queried
Archived

For an example of “contexts”, I mean that a DBA may be thinking about how this applies to data in a relational database. A DBRE may be thinking about how these tasks are automated across both on-prem and cloud environments, with multiple types of data stores. Meanwhile a DE may be thinking about how this applies in a Lakehouse or Event Streaming scenario.

In other words, everyone cares about compliance, data loss, security, and availability in one way or another.

DBAs, DBREs, and DEs all generally know and use SQL on a regular basis.

Database Administrator key strengths

Database Administrators are often highly knowledgeable regarding:

Relational databases (mostly OLTP)
Bespoke query tuning and performance troubleshooting
Scaling monoliths
Patching and upgrades (often semi-automated)
Environment standardization

These are the folks who are most likely to learn 1,001 configuration settings for a specific database technology, along with the pros and cons of using each setting, alone or in conjunction with one another.

While DBAs often implement automation, often writing bespoke scripts is a small part of their job and much of the focus is on adopting off-the-shelf or open source automation tools written by others. Establishing reliable processes is an emphasis and “protecting” data in source databases is an emphasis.

Database Reliability Engineer key strengths

The DBRE role focuses on automation as well as empowering developers to have shared ownership of code. While DBAs are generally “protectors” of data, DBREs have a mission of helping lay down safe and flexible guardrails for data that allows development to move faster and more flexibly.

DBREs key strengths are:

Data related infrastructure as code (IaC)
Automated cost monitoring, availability monitoring, performance monitoring at scale
Turning “pets” into “cattle”
Enabling developers to own and support their own data code
Data code versioning, testing, and automated deployments

In addition to SQL, DBREs use various command line and scripting languages for automation.

Data Engineer key strengths

The Data Engineer role often interfaces with Enterprise Reporting and Data Scientist teams. Data Engineers generally load, wrangle, and help manage vast amounts of data from a large variety of sources within an Enterprise.

You might want to be a Data Engineer if the following excite you:

Data lakes, lakehouses, warehouses: architecture and implementation
Data pipelines, transformations, data quality checks
Streaming architectures
Governance, metadata, lineage

Data Engineers often use Python quite a lot, sometimes more than they use SQL. Like DBREs, they have a focus on using version control and automating testing and deployments, although this is more related to notebooks, data pipelines, and transformation (as compared to Terraform or Bicep deployments and automating monitoring).

When interviewing, ask what these terms mean to the employer

I recently saw a conversation online between Data Engineers discussing that the Data Engineer role is defined quite differently at Meta than at Google, which can make for an awkward interview for candidates who assume that expectations for the role are similar.

This is true for any role, really. It’s always worth asking in a job interview: what are the primary pain points you want this person to solve? What are the key strengths that a candidate will need to succeed in this role?

What’s your take?

What have I missed? How are your definitions different? Tell me in the comments, or write your own post and share a link.

Should We Do Index Maintenance on an Azure SQL Managed Instance or Azure SQL Database?

help@KendraLittle.com (Kendra Little) — Mon, 17 Jul 2023 08:38:02 -0400

Have you ever received advice that was technically correct, but it was too hard to understand?

I think of this as “accidental bad advice,” because it can lead to confusion and bad outcomes. There’s a LOT of accidental bad advice out there on index maintenance for SQL Server and cloud versions like Azure SQL, even in the official documentation.

In this post I’m answering a common index maintenance question, and we’re going to keep it simple.

Do We Need Index Maintenance If We Have Fast Storage?

Last week I received a question about whether index maintenance was needed for an Azure SQL Managed instance or Azure SQL Database, given that the documentation says:

For most types of storage used in Azure SQL Database and Azure SQL Managed Instance, there is no difference in performance between sequential I/O and random I/O. This reduces the impact of index fragmentation on query performance.

Source: Optimize index maintenance to improve query performance and reduce resource consumption

Essentially, this is saying that often Azure SQL uses SSD for physical storage. It implies that therefore index maintenance isn’t very useful. A reader would be easily tempted to use this sentence to justify not setting up the maintenance at all.

Yes, We Need Index Maintenance, but Tune It to Do the Minimum

Even if you have very fast storage, if you don’t ever do index maintenance then you can end up with problems such as a lot of empty space in an index.

This is no big deal on small indexes (think less than 300MB to keep it simple) but can slow down performance on larger indexes.

In consulting it isn’t uncommon for me to find 20+ GB of empty space in larger databases when index maintenance hadn’t been implemented.
Think of this space as both disk space and space in memory. That means you can fit less in memory!
Memory is especially expensive in Azure SQL Managed Instance and Azure SQL Database. It’s kinda shocking how little you get for how much you pay, really.

But if you overdo index maintenance, then you are wasting CPU resources and can slow down concurrent activities.

So it’s a good general rule to have index maintenance, but to tune the maintenance to only do the minimum.

At this point, someone might say: “But Kendra, the documentation essentially says that after the quote!”

Sure, the official guidance says this. It says this so verbosely and in such excruciating detail that I must force myself to read the full page. Would I understand it if I didn’t have years of foundational knowledge on the topic? Probably not.

How Do We Make Index Maintenance Do the Minimum?

Like loads of folks, I’m a big fan of Ola Hallengren’s free SQL Server Index and Statistics Maintenance solution.

This tool should work for Azure SQL Managed Instance like it does for a SQL Server hosted on a VM, container, or server. It should also work on Amazon RDS.

For Azure SQL Database, “You need to download the objects as separate scripts” and come up with your own way to automate execution for them. (Source). One way to do this is via Azure Data Factory: Daniel Taylor explains how to set it up in this video.

Azure SQL supports online index operations for both Managed Instance and Database. Assuming your database may be queried during maintenance, here’s a sample command:

EXECUTE dbo.IndexOptimize
 @Databases = N'USER_DATABASES',
 @MinNumberOfPages = 40000,
 @FragmentationLow = NULL,
 @FragmentationMedium = N'INDEX_REORGANIZE, INDEX_REBUILD_ONLINE',
 @FragmentationHigh = N'INDEX_REBUILD_ONLINE, INDEX_REORGANIZE',
 @FragmentationLevel1 = 50,
 @FragmentationLevel2 = 80,
 @LogToTable = N'Y';

Let’s walk through these parameters at a high level. For more details, check Ola’s docs (which are great).

Databases – this will loop through all user databases. You can specify an individual database, use wildcards, and do much more.
MinNumberOfPages – this defaults to 1000 pages, or 7.85MB. That’s REAL small to me on modern storage. To skip smaller indexes, I suggest raising this up to 40000 pages, which is ~313MB.
FragmentationLevel1 and FragmentationLevel2 – I agree with Brent Ozar’s suggested limits of 50% and 80% as general starter levels.

Do You Have Index Maintenance Questions?

Feel free to ask away in the comments– small questions we can probably answer directly. Big questions might get their own post!

Even Smart People Make Terrible Data Modeling Decisions

help@KendraLittle.com (Kendra Little) — Tue, 11 Jul 2023 09:28:46 -0400

I was doing a bit of data analysis, and the resulting numbers didn’t quite add up.

I double-checked my queries. Had I goofed in my sql? Nope. Next, I looked into if some of the data was in an inconsistent state.

What I found was worse than what I’d imagined. As a data person, it made me feel sad and icky.

That’s because it’s usually not too hard to clean up bad data. It’s almost always much harder to fix a badly designed data model which is already established in production.

The Schema

I’m anonymizing the schema here, because it doesn’t matter who did this. Badly designed relational databases are very, very common– this one made me feel a certain way because it’s uniquely bad, but it’s certainly not the worst thing anyone’s ever done to a database. It was simply pernicious.

So let’s say we have a table with many millions of rows, named Widgets. Widgets is primarily a table in an OLTP database, the but database is also used for lots of analytics.

Widgets has many columns, but among them are:

WidgetId, a surrogate key that uniquely identifies every widget in the table.
WidgetTypeId, which joins to WidgetTypes.
WidgetOwnerId, which joins to WidgetOwner.

Here’s a mermaid.js entity relationship diagram:

erDiagram Widgets { WidgetId int PK WidgetTypeId int FK WidgetOwnerId int FK } WidgetTypes { WidgetTypeId int PK WidgetTypeName varchar(255) } WidgetOwner { WidgetOwnerId int PK WidgetOwnerName varchar(255) } WidgetTypes ||--|{ Widgets : "Has" WidgetOwner ||--|{ Widgets : "Has"

What Wasn’t Adding Up: Ownerless Widgets

Widgets was a very commonly used and core table for the OLTP database. What I found was that although the table had many millions of rows, all of them had a WidgetOwnerId… except for around six rows. Only six.

It smelled like a data cleansing problem at first, but when I had a look at the six rows I noticed something odd: they all had negative integers as the WidgetId, while every other row in the table had positive integer ids.

In other words:

A handful of widgets had a negative WidgetId, and all of these rows had a null WidgetOwnerId
Many millions of widgets each had a positive WidgetId, and all of these rows had a valid WidgetOwnerId
All widgets had a WidgetTypeId.

This seemed like a clue. I began asking around about what the negative WidgetId values meant.

The answer was that they were actually a very different type of widget altogether. But when that type of widget was implemented, a new WidgetType wasn’t created… instead they re-used a different (let’s call it “incorrect”) WidgetType, and used a negative identifier to indicate that this wasn’t any ordinary widget.

In other words, WidgetType was not only determined by the WidgetTypeId column in the table. To actually know the WidgetType, you also had to look at whether WidgetId was a positive or negative value.

What’s So Terrible About This? Let Us Count the Ways

Essentially, we’ve got “hidden” business logic here, and it’s business logic which doesn’t follow the established conventions of normalization. Here’s the top three negative impacts:

Bad/breaking code. This is a weird pattern that is not easy to notice. Anyone not intimately familiar with the data can easily make mistakes when working with this data, which could have significant impacts on applications using the OLTP database.

Extra cognitive load. This pattern is not easy to get comfortable with or to remember, especially if you’re not a data person. Having to teach everyone who works with this object the nuances of how it is designed, and have them remember how to work with it adds a weird complexity that is tough to remember.

Knock-on effects downstream. Want to put this table in a data warehouse? The choices are to leave the structure as is, in which case you risk incorrect analytics / extra cognitive load for your analysts. Or you could build in complex transformations that may be very fragile depending on how the object changes in the future. Neither of these is a great option.

The Root Cause Wasn’t Dumb People… It Was a Cultural Problem

To folks used to normalized data structures, this sounds pretty dumb. The tables were already normalized! Why make it weird?

Well, this didn’t happen because folks didn’t understand normalization. This happened as a “creative” solution to ship some changes faster by reusing code. That decision was driven by strong pressure from a leader to meet deadlines.

What became a technical problem was caused by a leader who created a problematic culture, in which folks felt like they didn’t have time to do the right thing. Instead they live in a perpetual “just force it to work NOW” moment. It turns out the codebase was full of weird hacks that had been made to meet shipping demands and schedules. And the hacks were growing. This was merely one example.

The result of a constant “just force it to work NOW” culture is:

An extremely fragile production environment that breaks easily– and even small errors cause full outages.
An engineering team that has a hard time hiring junior or even mid-level software developers because of the cognitive load required to understand and work with the complex code.
Frequent incorrect analytics.

This is not an easy situation to dig out of while also building new features, so any changes need to be made carefully, bit by bit. And you also have to do something about all those fresh anti-patterns entering the architecture on the regular to get those new features shipped.

My Performance Tuning Precon and Git Session at PASS Data Community Summit 2023

help@KendraLittle.com (Kendra Little) — Mon, 10 Jul 2023 10:11:37 -0400

I’m thrilled to be heading to Seattle in November for the PASS Data Community Summit. My favorite things about the PASS Summit are making connections, learning from folks, broading my horizons, helping build a vibrant community, and teaching.

Teaching is a great privilege, and I’m excited to be giving both a pre-conference session and a regular session. I chat about the sessions in this ~4 minute video.

My Precon - Performance Tuning Secrets: Learn From My Mistakes

I wanted to do something different for this day-long session, so I asked ChatGPT to generate an outline for a performance tuning session for SQL Server. The result was comprehensive, but it’s been done many times before. It felt… kinda boring. I’m flipping it upside down and building the session around my actual experiences, not from a list of concepts.

During this day, we’ll:

Delve into real-world stories and practical advice for tuning the SQL Server engine on-prem and in the cloud.
Cover essential concepts like wait stats and query plans, along with an array of free tools.
Approach these from a practical, real-world perspective, focusing on instances where I made mistakes or where alternative approaches might not have been apparent.

From years of perf tuning and teaching it, both in the classroom and in consulting situations, I’ve found that an emphasis on practicality is critical: it’s what will help you keep from getting lost in the weeds over and over again.

Database professionals need to adopt a strategic mindset, or we’ll end up constantly in reactive mode with a defensive mentality. While reactivity is important, we need to get ahead of potential issues and establish systems where risk is mitigated. Our goal is to create a safer and more stable environment, minimizing the need for constant reaction. While this is a straightforward concept, it’s very hard to understand what this means in reality without analyzing specific situations.

Moving out of a reactive mindset doesn’t mean limiting or compromising what we do. We’ll explore how to properly performance-tune and build robust solutions. It’s a challenging task, but a fun one, and we’re going to tackle it in this session.

My Regular Session: Git and GitHub for Database Folks

Version control is essential for automation and has become a fundamental skill in software development and IT. I use Git on a regular basis and, weirdly enough, I even find it fun.

In this session, we’ll cover the basics and build up the necessary skills. The good news is that learning these skills has never been easier. There are numerous helpful free and paid tools available to make using git more enjoyable and accessible than ever before.

I Drew a Comic

How to Register

Check out more about the conference and register at the PASS Data Community Summit site.

Hope to see you in Seattle in November!

Adding a Netlify Contact Form to a Hugo Static Site

help@KendraLittle.com (Kendra Little) — Thu, 29 Jun 2023 00:00:00 +0000

I redid my static site’s contact form with a free option from my host, Netlify.

Here’s why I changed my static site host and how I got a Netlify contact form to work with a blog built on Hugo and the Mainroad theme.

Why I Moved to Netlify

I wrote back in 2021 about Moving From WordPress to an Azure static site with Hugo. Since then, I migrated away from Azure Static Sites to Netlify’s free plan.

I moved to Netlify for a couple of reasons:

1. I was hitting storage quotas with the Azure static site that caused deployments to fail

I’d already reduced image file sizes where it made sense and moved larger files off to 3rd party services for hosting and embedding. There was little advice on troubleshooting this issue in Azure, and I have a lot of hobbies to tend to.

Netlify’s free plan doesn’t have this limitation.

2. Netlify offers some other cool things on the free plan, such as the ability to use interactive forms hosted on their servers

While I’d previously set up a simple option embedding a Google form on my contact page, I didn’t love the workflow with this solution. I wanted something better.

I probably should have blogged about how I migrated to Netlify, if only because it was so simple that I barely remember what I did. I remember that Netlify walked me through everything, and that it was quite easy. I don’t think I even had to provide a credit card number in the process.

It took me a while to get around to redoing my contact form, and here’s how I did it.

Enable Form Detection in Netlify

First, you’ll need to enable forms in your Netlify site configuration.

Sign into the Netlify UI, then go to Site configuration -> Forms.
Select Enable form detection.

This won’t do anything right away. The next time you deploy the site, Netlify will look for forms which want to hook up.

Start Serving Your Site Locally

From a terminal, I ran: hugo serve

I then regularly referred to my locally served version of the site as I stepped through the following. This helped me check functionality as I saved changes to ensure I knew what was working.

Create a Hugo HTML Layout for Your Contact Page Form

While there are several posts and tutorials out there on how to do this, I hadn’t worked with Hugo in a minute. I had to spend some time remembering how this all works, and which files I even use for html. Oh, right, the magic happens in Go templates!

Under my layouts folder, I added a new folder, contact. In this folder I created a new file named single.html.

graph LR C["📁 content"] --> layouts["📁 layouts"] --> single-html["single.html"]

{{ define "main" }}
<main class="main" role="main">

 <form name="contact" class="contact-form width-normal" action="/contact-thank-you/" method="POST" data-netlify="true" netlify-honeypot="bot-field">
 <input type="hidden" name="form-name" value="contact" />
 <p class="hidden">
 <label><input name="bot-field" /></label>
 </p>
 <div class="form-group">
 <!-- Text input-->
 <label class="contact-form-1 control-label" for="Name"></label>
 <div class="contact-form-1">
 <input id="contact-form-name" name="Name" type="text" placeholder="Name" class="form-control input-md" required="" autocomplete="off" maxlength="100">
 </div>
 <!-- Text input-->
 <label class="contact-form-1 control-label" for="Email"></label>
 <div class="contact-form-1">
 <input id="contact-form-email" name="Email" type="email" placeholder="Email" class="form-control input-md" required="" autocomplete="off" maxlength="100">
 </div>
 <!-- Text input-->
 <label class="contact-form-1 control-label" for="Subject"></label>
 <div class="contact-form-1">
 <input id="contact-form-subject" name="Subject" type="text" placeholder="Subject" class="form-control input-md" required="" autocomplete="off" maxlength="200">
 </div>
 <!-- Textarea -->
 <label class="contact-form-1 control-label" for=""></label>
 <textarea class="form-control" id="contact-form-message" name="Message" placeholder="What's up?" rows="8"></textarea>
 <!-- Button -->
 <button class="contact-button" type="submit" value="Submit" id="Form-submit">Submit</button>
 </div>
 </form>
</main>

{{ partial "authorbox.html" . }}
{{ partial "pager.html" . }}

{{ end }}

Some Notes on this Form

The form opening tag specifies data-netlify="true". This means that the next time you publish the site, the form may be detected by Netlify, as long as you’ve enabled forms on the site.
The form opening tag specifies netlify-honeypot="bot-field". There is also an input named bot-field that has a css class specified named hidden. Together, these are used as a spam reduction feature in Netlify.

At this point, as long as you don’t already have a contact folder which this template would apply to, you shouldn’t notice any differences in your locally served copy of your site.

Create a Contact Page That References the Layout

If I was adding a new contact page, I could have added a new folder named content/contact. This would automatically pick up the contact template.

However, I already had a contact page in place with a URL I wanted to maintain that didn’t involve a “contact” subfolder, so I edited my existing file which I had under content/contact-kendra-little.md.

graph LR C["📁 content"] --> contact["📁 contact"] --> contactkendra["contact-kendra-little.md"]

The entire definition of this page is as follows. The important bit is that I added the line type: contact to the front matter so that it will pick up that layout.

---
title: "Contact"
draft: false
authorbox: false
pager: false
menu: main
type: contact
---

At this point, your locally served copy of the site should have a contact form which displays the html form in the template.

The styling, however, may look a little wonky. Mine sure did. And if you don’t have any css already that hides fields with a class named hidden, you will also see the honeypot field on your form, which will look odd.

Add CSS Styling as Needed

The next thing I had to remember was where to customize css files. At this point, I asked GitHub Copilot for help:

I already had a file named static/css/custom.css, but create one if needed.

graph LR C["📁 content"] --> static["📁 static"] --> css["📁 css"] --> custom-css["custom.css"]

I added the following css to get my form working, using my locally served copy of the site as I made changes.

 .hidden {
 display: none;
 }

 .contact-form-1 input[type="text"],
 .contact-form-1 input[type="email"],
 textarea {
 font-family:'Lucida Sans', 'Lucida Sans Regular', 'Lucida Grande', 'Lucida Sans Unicode', Geneva, Verdana, sans-serif;
 font-size: 1rem;
 }

 /* Add a background color and padding */
 .contact-form-1 {
 max-width: 90%;
 padding-top: 6px;
 padding-bottom: 6px;
 border-radius: 5px;
 }

 /* Style the submit button */
 .contact-button {
 background-color: #ff8400;
 color: white;
 padding: 4px 4px;
 border: none;
 border-radius: 4px;
 cursor: pointer;
 }

.contact-button:hover {
 background-color: #ff5100;
 }

Create a Thank-You Page (Optional)

My form declaration specifies action="/contact-thank-you/".

graph LR C["📁 content"] --> custom-css["contact-thank-you.md"]

I created a page at content/contact-thank-you.md with the following definition:

---
title: "Thanks for getting in touch"
draft: false
authorbox: false
pager: false
---

I try to reply to most emails within a few business days.

{{< figure src="/images/contact-kendra-little.jpg" width="650" >}}

Commit Your Changes and Deploy the Site

To actually test the form, you need to commit your changes and deploy your site. You can do this a bunch of ways with Netlify– go with whatever you are comfortable with, and which meets the requirements of your site.

Configure Email Notifications for Form Submissions

I want to get an email when someone fills out the contact form on my site. You can configure this in the Netlify UI by enabling email notifications for one or more forms. Essentially:

Log into Netlify.
Go to Site configuration -> Forms -> Form notifications.
Select Add notification, then step through the wizard to configure the notification.

Test and Troubleshoot

Now test out using the form on your published site, and make sure you get the email.

Netlify provides troubleshooting tips if you run into problems.

I used the following sources as references when setting this up:

Is It OK to Show My Database Schema to ChatGPT and GitHub Copilot?

help@KendraLittle.com (Kendra Little) — Fri, 23 Jun 2023 10:12:56 -0400

I’m answering two questions from Brent Ozar’s list of user questions open for answers.

Q: What’s your opinion of entering confidential info in chat gpt? Will we see AI therapist chat bots?

Q: In terms of security, is it OK to expose your database to tools like GitHub Copilot in Azure Data Studio? Someone will know that your email address column is not encrypted or a stored procedure is not parsing its input parameters when dynamic T-SQL is built.

In general, it’s best to assume that any data you enter into an online website or into an extension or tool may be stored, used, shared, or sold– until you find documentation that explicitly promises otherwise. This is true both for generative AI tools as well as any other tooling.

It’s also best to chat with your leadership about this. Many companies have very specific policies about what you can and cannot do with company data.

ChatGPT and OpenAI

Don’t enter someone else’s confidential info into the ChatGPT website. Don’t enter your own confidential info into the ChatGPT website unless you want it to no longer be confidential. The privacy policy and related article, How ChatGPT and Our Language Models Are Developed go into more information about how data shared with ChatGPT’s website is used.

Note that the ChatGPT website is not the same as the Azure Open AI service. These are different tools, and they have different privacy policies. For the Azure Open AI Service, the article Data, privacy, and security for Azure OpenAI service covers what data is processed and how it is used.

It is possible to use the Open AI service under a data agreement that opts you out of the logging and human review process, but you need to specifically arrange that. In other words, it is possible for applications to be built that can use even very sensitive, highly confidential, or tightly regulated data safely with these services.

GitHub Copilot

If you’re using GitHub Copilot for business, check out the GitHub Copilot for Business Privacy Statement.

The statement explains that snippets of your code that are used to provide suggestions are discarded after suggestions are returned, and are not retained. It does collect some telemetry data regarding how you use the IDE and editor along with general usage data.

GitHub Copilot has now been around long enough that many software development companies have reviewed the product and created a policy about whether or not it should be used by the organization, so checking with your management can save you some time especially in this case.

What About All These Other “Copilots”?

Lots of companies, including Microsoft, are introducing assistants and copilots throughout their platforms.

I’m sorry, but we’re gonna get to check the data privacy statements for each one, friends. Here’s an example for the Microsoft Power Platform.

AI Therapists Have Been Around for a While

AI Therapist chatbots already exist, and have for a while, actually!

One I like is “Woebot” – https://woebothealth.com/. Woebot is a free app (at least for now, has been for a while though) that specializes in cognitive behavioral therapy style conversations.

The Woebot FAQ talks a bit about how the data is stored and options to delete it, and the privacy policy specifies under what conditions they will share data with third parties.

When NOT to Choose SQL Server to Store Your Data

help@KendraLittle.com (Kendra Little) — Wed, 21 Jun 2023 12:29:57 -0400

I recently posted on LinkedIn that I was interviewing for a SQL Server database administrator role for the first time in a long time. I invited folks: “Ask me your favorite interview question!”

Stephen Vakil had a great one: “when should you use SQL Server to store your data?”

For situations where SQL Server is already in use and there’s a relatively low barrier to entry, I think it’s simpler to turn this question around and ask, “When is SQL Server not a good choice for storing your data?”

This post is also available as a podcast episode, and on YouTube.

You Don’t Actually Need a Database

This is going to seem like an obvious question but…. do you actually need a database?

Would flat files work for your purpose? Do you only need a readable cache and very low latency? Would an API interface be better? (And maybe what’s behind it is a database, or maybe it’s not.)

It’s very easy to get comfortable with a database that the team is quite familiar with, and to default to using it. Especially because of this, it’s a good practice to as regularly: do we need this in a database? Is there a less expensive and easier to manage option, or an architecture that would work better in the long run?

You’re Not Using Structured Data

SQL Server’s primary strength is being a reliable, high performing relational database that handles structured data well. This means it’s great for situations where:

You have or can develop a schema with fixed columns that have suitable data types. (Aka “structured data”.)
You want to apply some level of normalization to the data to reduce duplication and enforce integrity.

This is the core scenario where the SQL Server database engine and query optimization shine.

While SQL Server can deal with unstructured data, if this is the primary use case for the application then a type of database that is designed to optimize storing and accessing unstructured data is worth considering. Databases that specialize in handling unstructured data include MongoDB and Couchbase, but the right choice will vary by your use case, your deployment scenario (cloud/on-prem), and other variables considered below.

If you are not interested in normalizing data in SQL Server because you wish to use the database primarily for analytics, you might also want to consider other databases…

You Don’t Have an OLTP or Mixed Workload - You Want Only Analytics, for Example

SQL Server is very good at transactional processing, with lots of inserts, updates, and deletes. SQL Server has multiple options on how to implement ACID compliant transactions when you have a workload that consists of concurrent reads and writes. Here’s a review of those ACID properties:

Atomic - transactions are “all or nothing,” meaning they complete in their entirety or roll back in their entirety.
Consistent - transactions must follow all implemented database constraints or business logic implemented by structures in the database. For example, if a transaction violates a unique constraint, it must fail and roll back in its entirety.
Isolated - transactions shouldn’t interfere with one another if running simultaneously – or at least, they should only interfere with one another based on what you have set as your isolation level.
Durable - transactions can’t vanish after they commit, even if there is a power or system failure. This means that the transaction information must be stored in a non-volatile way (think writing to disk). Write-ahead logging (WAL) helps achieve this in SQL Server.

If your primary use case for SQL Server is an OLTP workload, but you’d also like to do some reporting against those tables, that can sometimes work well in SQL Server: it even has a columnar storage option that can help with large scans.

However, if your primary or only use case for a database is reporting or analytics, you might not want to use “classic” SQL Server. Depending on your use case, you might consider a database or warehousing approach designed purely for analytics. There are many different options for analytics workloads, but here are three that come up a lot these days:

Lakehouse approaches, which combine data lake and data warehouse components, are increasingly popular, especially when you have large amounts of raw data. Databricks is a leading vendor in this space.
Columnar databases, such as Snowflake, have been developed specifically to optimize analytics workloads within a PAAS offering. When there is a desire to have SQL “just go fast” with minimal administration for structured data, these specialized databases shine.
Vector databases enable very fast search and retrieval of data based on its similarity or relevance to other data. These are very popular for analytics related to Natural Language Processing (NLP).

You Need to Write to Multiple Nodes for a Single Database

Technically, SQL Server has an option for having multiple writeable copies of a database: peer-to-peer replication.

But I haven’t heard of anyone deploying a new peer-to-peer setup in many years. It’s not a popular choice, because it tends to be pretty fragile: you need to design the schema carefully to avoid having a lot of conflicts if the same rows are updated on different nodes. Microsoft recommends:

To avoid potential data inconsistency, make sure that you avoid conflicts in a peer-to-peer topology, even with conflict detection enabled. To ensure that write operations for a particular row are performed at only one node, applications that access and change data must partition insert, update, and delete operations. This partitioning ensures that modifications to a given row originating at one node are synchronized with all other nodes in the topology before the row is modified by a different node.

Essentially, some very specific application designs may work with peer to peer replication, but even then, living with P2P and troubleshooting it in real-world scenarios is quite complex and time consuming. I would generally look at other options first if I need to scale out writes for a single database.

Other options for SQL Server, including Azure SQL variations of SQL Server, allow only one writable node for a database.

Your Team Wants a PaaS Solution

This is just a terminology thing– “SQL Server” generally refers to the “boxed software” version of SQL Server that you either install yourself, or run pre-installed on a VM in an implementation like SQL Server on Azure VM.

SQL Server can require quite a lot of care and feeding: backups, maintenance, configuration, upgrades/patching, and monitoring. By using a Platform as a Service (PaaS) option, you can limit some of this work.

For these cases, a hosted database option can be the right answer. This might be something very similar to SQL Server, like Azure SQL Database, Azure SQL Managed Instance, or Amazon RDS for SQL Server. Or a hosted version of another relational database, based on your preference, might fit, depending on the requirements of the situation .

Your Team Needs a Cheap Solution

SQL Server licensing gets expensive quickly. Sometimes this leads to people just throwing things on the SQL Server: hey, we paid a lot for these licenses, might as well use it!

And while that’s fine in many cases, eventually adding on all those use cases will likely lead to spending more on SQL Server licensing. Cost is generally a consideration for any project, and if you need a cheap solution then SQL Server generally shouldn’t be your first choice. (Unless, of course, you work at a place that uses a lot of Oracle. In that case, the SQL Server will look cheap.)

Career Navigation in the Time of Tech Layoffs

help@KendraLittle.com (Kendra Little) — Tue, 23 May 2023 00:00:00 +0000

Reorganizations and layoffs are now commonplace in the tech industry: a daily occurrence. It’s increasingly necessary for tech workers mentally prepare for potential job loss and uncertainties that arise. It’s a common antipattern to believe that it’s “too late” to prepare for a layoff or unexpected reorganization if a layoff has already occurred in your organization.

It’s never too late, and it’s always the right time these days to start mentally preparing yourself for change– because there’s reliably another change around the corner.

The Reality of Tech Layoffs

Even tech companies raking in millions or billions in profits for shareholders are cutting costs, laying off workers, and talking about reductions in force. The phrases “economic headwinds” and “I take full responsibility” are both doing a massive amount of work, and these repeated excuses contribute to an environment of uncertainty. The hype cycle over AI and rapid changes in technical capabilities also contribute to erratic executive decisions and tech worker worries. Acknowledging the stress and challenges that come with these situations is helpful for preparing ourselves mentally.

Navigating through layoffs, reorganizations, and technical evolutionary moments requires a resilient mindset. It’s important to acknowledge the potential impact on our careers and emotions. Uncertainty can breed anxiety and fear, but being mentally prepared helps us approach the situation with a sense of control and adaptability.

I remind myself often that change is inevitable, and that sometimes changes that start with a lot of discomfort lead to opportunities. I encourage myself to focus on building my powers of resilience, which I think of as a type of mental ‘muscle’.

Lots of Us Don’t Have a Detailed Plan for What to Do Next

I recently ran a poll among my LinkedIn network and asked how folks are approaching this time. With 194 respondents, the results included:

24% of respondents are “hoping for the best,” indicating that they may not have a specific career direction in mind but are optimistic about finding opportunities.
37% of respondents have a general direction they know they want to go in but haven’t yet developed a detailed plan. This group recognizes the need for change but requires further clarity.

More than 50% of us don’t have a basic or detailed plan for what to do if our job is suddenly eliminated.

I don’t think that everyone needs a detailed plan. I do think that there are some steps that we all can take that will make our lives easier, however, especially if we aren’t in a situation where we are ready to form a detailed plan.

I personally am in the “general direction” bucket, because I think I don’t want my next job to have the same job title as I have now – I think for my next job I probably want to do something a bit different from Technical Product Management. I’m not 100% what that looks like, but I will benefit from doing what I can to explore my future interests now, even if at a high level.

Revive or Build Your Network

For many of us, building connections and reviving our professional network helps orient us mentally and can be a first step to improving mental resilience: it helps us feel less isolated.

Consider reconnecting with and seeking support from trusted colleagues, mentors, and your professional networks. Connecting with others who are experiencing similar situations can provide valuable insights and emotional support, helping you navigate through the uncertainty with greater confidence.

If, like me, you are considering of shifting your career direction, this can be a great time to set up informational interviews (which can be done synchronously or through chat or email) with people you respect to ask about roles which you find interesting. You can ask for advice: what makes people excellent in this role? What are the table stakes for getting into this type of role? What would the person advise someone with your background and skills to do to move into that type of role in the future?

These conversations can not only help inform you and give you a greater sense of direction, but they also build your network and can revive past connections that have faded.

Consider a Flexible Career Plan

Having a general career direction can provide a sense of stability. A flexible career plan allows you to adapt a bit where you are, and can also offer benefits for your current employer.

In a time of rapid change, layoffs, and reorgs, there are often ways to seek opportunities within your current job to develop skills that align with your desired future role. The trick here is that your manager is generally not going to do this for you: this is usually something you need to think of and pitch for yourself. You may need to try multiple times to suggest changes in how you work for it to stick, as well.

Think about areas where you want to grow and how you can leverage available resources, such as training programs or cross-functional projects. Perhaps brainstorm with a colleague how you can do things in a different way that is a benefit to your team and company as well as yourself.

Avoid the Trap of Overworking

A common antipattern that folks fall into in a time of “economic headwinds” is believing that working excessive hours is the ultimate shield against layoffs. Not only is this untrue in my experience, it’s also bad for your mental resilience.

Workforce reductions often do NOT spare those who work the hardest or the longest hours, for various reasons. Overworking increases your risk of burnout and takes a toll on your health. Focusing on limiting your time working and in building mental resilience and your network will help you more in the long run.

ChatGPT Says SSMS Is the Best SQL Server Monitoring Tool

help@KendraLittle.com (Kendra Little) — Wed, 03 May 2023 08:27:18 -0400

One thing I enjoy about AI chatbots is that they can help me recognize when I’ve missed something obvious.

The Question

I prompted ChatGPT (free version) with: “What are the top 10 products to monitor sql server databases? List the top three features and drawbacks of each, give detailed contrasts/comparisons of the top 3 by popularity.”

It responded with a list, and the top winner of this popularity contest was: SQL Server Management Studio (SSMS).

My Initial Reaction

My first thought was, “that’s not a monitoring tool!”

What the Robot Said

But I was thinking about SSMS and my question incorrectly. ChatGPT pointed out that SSMS’ top features are that it’s free, included with SQL Server, has an extensive toolset, and is easy to use. The drawbacks are that it has limited automation, lacks real-time monitoring, can be slow on large databases. At the end of the list of tools, ChatGPT suggested that, “the best SQL server monitoring tool for your organization will depend on the size and complexity of your databases, your budget, and your custom monitoring requirements.” And that’s absolutely true.

What I Learned

This uncovered an assumption I hadn’t realized that I hold: when I use the term “monitoring,” I assume that monitoring is automated. This is only a subset of monitoring, however. In many environments, it may be acceptable for monitoring to be manual, depending on the factors listed above. And when there’s no budget and very limited monitoring requirements, SSMS is a fine choice!

As we enter the world of the Rising Chatbots, I use these tools more and more. I don’t trust that the tools output accurate information, but rather that they might lead me in the direction of useful information or output, as long as I put in the due diligence needed to use it wisely.

Use ChatGPT to See Multiple Perspectives: An Example With ORMs and Database Code

help@KendraLittle.com (Kendra Little) — Tue, 24 Jan 2023 00:00:00 +0000

There’s a lot of hype, concern, and fear regarding generative AI lately. Tools like ChatGPT are so good at generating groups of words that it feels like magic– however, generative AI doesn’t have the ability to understand or verify the language it generates. For example, it’s been used to write news stories for CNET – but sometimes included facts that are just plain wrong.

One positive use case of a generative AI tool like ChatGPT is to ask it to make a case for different perspectives than your own, and use this as a starting point to broaden your understanding – with the knowledge that we need to verify everything ChatGPT tells us. As an example, let’s chat to ChatGPT about Object Relational Mapping tools (ORMs).

Prompt: Give Arguments for and Against Using an ORM to Generate Database Code

As someone who generally sits closer to database code than to application code, I’ve often wondered, “Why do developers love ORMs so much?” I get the basics: it can make getting started a lot faster. But since I haven’t actually used ORMs hands-on, there’s plenty I can learn.

To start out, I asked ChatGPT to argue for and against using an ORM, specifically in the context of generating database code.

ChatGPT responded:

Arguments for using an ORM to generate database code:

Abstraction: ORMs provide an abstraction layer between the application and the database, making it easier for developers to interact with the database without needing to know the underlying SQL.

Productivity: ORMs can automatically generate common database operations, such as CRUD operations, which can save time and increase productivity for developers.

Portability: ORMs can allow developers to work with multiple databases by providing a consistent interface, making it easier to switch between different databases or to change the underlying database technology.

Security: ORMs can help prevent SQL injection attacks by automatically parameterizing queries and escaping user input.

Maintenance: ORMs can automatically handle database schema changes, making it easier to maintain the application over time.

Arguments against using an ORM to generate database code:

Performance: ORMs can add an additional layer of overhead and may negatively impact performance, especially for complex or large datasets.

Complexity: ORMs can add complexity to the codebase, making it harder to troubleshoot and maintain the application.

Flexibility: ORMs may not provide the same level of flexibility as writing raw SQL, making it harder to optimize performance or take advantage of specific features of a particular database.

Learning curve: ORMs can have a steep learning curve for developers who are not familiar with them, making it harder for new developers to understand the codebase.

Obscurity: ORMs can obscure the underlying database operations, making it harder for developers to understand how the application is interacting with the database.

I’m more familiar with the arguments against using ORMs to generate database code, but I was still pretty impressed with this as a general summary of two different perspectives.

Prompt: What Are the Challenges With Combining an ORM With Custom Stored Procedures?

Digging in a little, one thing that can be frustrating is hitting a performance problem when an ORM is generating a bad query. One can often rewrite the query and it will perform much better, but then you hit the question: how do you tell the ORM to use different syntax? One option is to combine the ORM with procedures.

A lot of folks say this isn’t always doable and introduces problems. I asked ChatGPT what those problems might be.

ChatGPT responded:

Integration: Integrating ORMs with custom stored procedures can be challenging, as the ORM may not be designed to work seamlessly with stored procedures, and the developer may need to write additional code to handle the integration.

Mapping: ORMs rely on a mapping between the application’s data model and the database schema, and stored procedures can make this mapping more complex, as the ORM may not be able to automatically map the output of a stored procedure to the application’s data model.

Performance: Stored procedures can provide better performance than dynamically generated SQL in some cases, but executing stored procedures through an ORM can add an additional layer of overhead and may negatively impact performance.

Debugging: Debugging issues that arise from the integration of ORMs and stored procedures can be challenging, as the ORM may not provide detailed information about the execution of stored procedures.

Maintenance: Stored procedures can make it harder to maintain the application over time, as changes to the stored procedures may require changes to the ORM’s codebase, and vice versa.

Portability: ORMs allow to work with multiple databases, but if your application relies on a lot of stored procedures, it may limit the ability to switch between different databases or to change the underlying database technology without rewriting the stored procedures.

Prompt: Are There Tools or Processes That Can Make It Easier to Combine ORMs With Other Programming Models for Database Code?

Confession: I don’t think I’ve ever asked a human this. I probably should have, but for various reasons I never had time to fit it into a conversation. So I asked a language-generating web page.

ChatGPT responded:

Code Generation: Some ORMs include code generation tools that can automatically generate code for stored procedures, views, and other database objects. This can make it easier to integrate the ORM with existing database code.

Data Access Layers: Creating a data access layer (DAL) that sits between the ORM and the application code can make it easier to handle the integration of the ORM with other programming models. The DAL can handle the execution of stored procedures and other custom database code, and provide a consistent interface for the application code to interact with the ORM.

Stored Procedures Wrappers: Some ORM frameworks provide wrappers that allow developers to call stored procedures as if they were regular functions, this way the ORM can handle the integration and provide a consistent interface for the application.

ORM-agnostic libraries: Some libraries can be used to provide a consistent interface for working with databases, regardless of the ORM in use. This can make it easier to switch between ORMs or to work with multiple ORMs in the same codebase.

Third-party libraries: There are third-party libraries that can provide additional functionality to the ORM like stored procedures support, this way the ORM can provide the abstraction layer and the third-party library can provide the stored procedures support.

It’s important to note that these tools and processes can vary based on the specific ORM and programming models you are using, and some may be more effective than others depending on the requirements of your application.

Hmmm, it seems like there’s no easy answers, but these pointers are interesting things that I could explore in a real world problem solving situation.

Lastly, I considered that technology, including ORMs change fast, and improvements are often made. I asked ChatGPT how ORM tooling has improved over time when it comes to generating database code.

ChatGPT responded:

Over time, ORM tools have undergone various improvements related to database code, some of the major improvements include:

Performance: ORMs have become more efficient at generating and executing database code, and many now include features such as caching and lazy loading to improve performance.

Support for multiple databases: ORMs have become more flexible and now support a wide range of databases, this allows developers to work with multiple databases using the same ORM.

Support for stored procedures and views: ORMs have improved their support for stored procedures and views, allowing developers to use these database objects in a more seamless way.

Better support for complex queries: ORMs have improved their ability to handle complex queries and joins, allowing developers to write more sophisticated queries with the ORM.

Improved support for migrations: ORMs have improved their support for database migrations, allowing developers to easily make changes to the database schema over time.

Better support for NoSQL databases: ORMs have improved their support for NoSQL databases, allowing developers to use ORMs to work with NoSQL databases in a similar way to how they would work with traditional relational databases.

More advanced features: ORMs have added more advanced features like multi-tenancy, sharding, and other features that are specific to certain databases.

Better integration with other technologies: ORMs have improved their integration with other technologies like caching systems, search engines, and queues, allowing developers to leverage these technologies in a more seamless way.

Better support for testing: ORMs have improved their support for testing, allowing developers to create test data and perform testing in a more efficient way.

It’s important to note that these improvements have happened over time, and are not all present in all ORM tools, and some ORM may have more or less improvements in certain areas.

Tools like ChatGPT (and tools like ORMs) don’t understand things like humans do. They don’t imagine things, they don’t lie, they just … make stuff up. Sometimes it can go very badly.

We need to use them carefully, and we need to work to improve them. But maybe, if we use them carefully, and if we focus on learning from humans, checking our facts, and looking for our blind spots, they can help us learn and grow.

3 Lessons Learned About Azure SQL Database

help@KendraLittle.com (Kendra Little) — Sun, 15 Jan 2023 13:02:14 -0500

Hosted cloud databases make a lot of administrative tasks easier, or take care of them for you altogether.

But here are three things that I’ve found a little too easy to forget about Azure SQL Database.

1. “Service Objective” Means “Compute Size”

When I think about service objectives, I tend to assume we’re talking about guaranteed uptime, often expressed in some number of 9’s.

But Azure SQL Database has its own language. The best resource I know of to learn that language is the Azure SQL glossary of terms.

The glossary defines service objective this way:

Compute size (service objective) is the amount of CPU, memory, and storage resources available for a single database or elastic pool. Compute size also defines resource consumption limits, such as maximum IOPS, maximum log rate, etc.

When you are choosing your compute size, you are generally not presented with a slider that shows the amount of memory your database will get. You need to take some extra effort to look it up in the docs. This is worth doing, because…

2. Database Memory Is Pretty Darn Limited, and Serverless Compute Has Even Less Memory

When you configure an Azure SQL Database in the Azure Portal, you choose a lot of things — but you never choose directly how much memory your database has. The amount of memory allocated to your database is determined by the pricing model and other options you choose.

If the database memory amount is displayed anywhere in the portal, I haven’t found it yet. I believe you have to know to go look in the documentation for resource limits, which are listed separately for the vCore purchasing model and the DTU purchasing model.

For example, as of this writing, under the vCore purchasing model using the General Purpose service tier:

8 vCores with Serverless compute (standard series Gen5) gets 24 GB max memory
8 vCores with Provisioned compute (standard series Gen5) gets 41.5 GB max memory

Do both of these memory limits seem terribly small to me for 8 vCores? Yes.

I suspect that if you say you want to increase your cache size only without adjusting other elements of compute, the Azure answer is that you should look into implementing something like a Redis cache. :shrug:

3. There Ain’t Simple Autogrowth Here, Friend, and an Application May Not Throw the Correct Error When You’re Out of Space

I remember when I first started as a DBA, early on my team hit an issue where an application was failing and we needed to figure out why. We determined that the database files were out of space. It took time to discover because the error message that bubbled up in the application was completely unrelated and was a red herring.

Some things never change, especially the “misleading error message” part.

Your Azure SQL Database settings control the maximum size for your database. Increasing the max size within your current service level object can be relatively cheap, compared to increasing vCore count, but it’s still something you need to manage yourself: there isn’t a simple “enable autogrowth” checkbox.

It’s worth setting up database monitoring and configuring alerts for the amount of space used / free space in your database files. You should also periodically review the maximum data size allowed for your SLO (vCore / DTU) in the resource limits and compare this to projections of how much space you’ll likely need for a time period, in case you’re at risk of needing more space than your configuration allows: upgrading your core count is generally pricier than moving the database storage size slider.

New Article on Performance Tuning with the Missing Indexes Feature in SQL Server

help@KendraLittle.com (Kendra Little) — Fri, 18 Mar 2022 08:06:28 -0400

We’ve just published a new article in the SQL docs, Tune nonclustered indexes with missing index suggestions . The article explains what the missing index feature is, limitations of the feature, and how to use missing index DMVs and missing index suggestions in Query Store to tune indexes.

Why Now?

The missing indexes feature has been around in SQL Server since (looks at watch) SQL Server 2005. Seventeen years later, we’ve added a new article on using the feature. Why?

Our partners in Microsoft Customer Service and Support let us know that we needed more guidance for customers in the documentation on using missing indexes. (Thanks to Joseph Pilov.)
I found that older guidance on using the feature had been archived. (How did I find the archived content? A link from one of my old blog posts. Blogging is forever my long-term memory.)
The older guidance was good, but didn’t mention some of the gotchas that can come with working with missing index requests: things like only one missing index request scripting from a query plan in SSMS, even if multiple requests were present in the query plan XML.
New features like Query Store have been released since the archived guidance was written. Query Store helps persist missing index requests across index maintenance, restarts/failovers, recompiles, and memory pressure.

All this considered, it made sense to use the archived guidance as a reference to put together a new article on using missing index requests to tune queries.

What Else May Come?

I’d like to write a complimentary article on tuning indexes with the Query Store. This will be an example of looking at a workload in Query Store, and using information both from missing indexes and from the execution plans to tune queries.

Learn More

Coincidentally, Erik Darling also published an article on missing indexes this week. Check out Understand Your Plan: Missing Index Requests.

Got Suggestions for the Missing Indexes Article?

If you see something you’d like to change on the missing indexes article, there are two options to share it:

Scroll down to the bottom of the article to the “Submit and view feedback” box. Select “This page”, and fill out an issue with your suggestion.
Make a suggestion in a pull request. To learn how, see How to contribute to SQL Server documentation.

Find Query Execution Timeouts with Query Store

help@KendraLittle.com (Kendra Little) — Fri, 11 Mar 2022 09:01:08 -0500

During a discussion of troubleshooting query timeouts in Azure SQL Database recently, I wondered: can you find queries that timed out in Query Store?

You can.

Query Timeout Essentials

Here are some basics regarding query timeouts for SQL Server, Azure SQL Database, and Azure SQL Managed Instance:

Execution timeouts are NOT set or enforced by the database engine. It is happy to wait for your query to run or for your transaction to be open for hours, days, or years, until you commit, cancel, or get disconnected.
Timeouts are set by the calling application. If you don’t explicitly set a timeout on your connection string, there is generally a default timeout that is used. Often, this is 30 seconds for query execution timeouts.
Connection timeouts are different than execution timeouts. If you see a connection timeout, often this is a firewall or network issue preventing you from connecting to the database (or perhaps it’s even offline). Execution timeouts are when the calling application doesn’t want to wait any more for a query to complete.

The Traditional Way of Finding Query Timeouts

If you start seeing query execution timeouts from an application, you don’t always get information about exactly what query had the problem in the error. The first steps are often to identify that query.

The “classic” way to do this is to run a trace against your database, using either classic SQL Trace (perhaps with Profiler), or an Extended Events trace. You trace to find an Attention event, which is also known as Error 3617.

This isn’t a bad way to go about things, but it’s time consuming and inconvenient: you generally only start tracing after you get a timeout error, and then you need to wait for the issue to happen again.

Finding Timeouts in Query Store

If you’ve enabled Query Store, you may be able to find timeouts immediately, without running a trace.

There’s a caveat here: Query Store doesn’t necessarily collect every query. The QUERY_CAPTURE_MODE setting may be set to AUTO, which works to balance the space taken up by plans and doesn’t necessarily capture all queries.

Let’s cause a quick timeout and view it in Query Store.

Reproducing a Query Execution Timeout

SQL Server Management Studio conveniently lets you set an execution timeout in the session options when you connect to a database. We’re going to take advantage of this to reproduce an execution timeout. I’m using the ContosoRetailDW database for this example, but you can adapt this pattern to any database suitable for testing (where blocking and possible data loss are OK).

I’ve enabled Query Store on the database with the following options:

ALTER DATABASE
 [ContosoRetailDW]
SET QUERY_STORE = ON
(
 OPERATION_MODE = READ_WRITE,
 CLEANUP_POLICY = (STALE_QUERY_THRESHOLD_DAYS = 90),
 DATA_FLUSH_INTERVAL_SECONDS = 900,
 MAX_STORAGE_SIZE_MB = 1024,
 INTERVAL_LENGTH_MINUTES = 60,
 SIZE_BASED_CLEANUP_MODE = AUTO,
 MAX_PLANS_PER_QUERY = 200,
 WAIT_STATS_CAPTURE_MODE = ON,
 QUERY_CAPTURE_MODE = ALL
);

ALL may not be the best QUERY_CAPTURE_MODE for production databases: in this case, it’s useful when I don’t have a real workload running. I’ve also raised the database compat level of the database, because this puppy is super old and I want to use TRY_CONVERT syntax in a query later.

ALTER DATABASE
 ContosoRetailDW
SET COMPATIBILITY_LEVEL = 140;
GO

First, connect a session to the database. Don’t mess with the execution timeouts on this session, leave it at unlimited. Run a command like the following:

BEGIN TRANSACTION;

DECLARE
 @OnlineSalesKey integer;

SELECT
 @OnlineSalesKey = fos.OnlineSalesKey
FROM dbo.FactOnlineSales AS fos
WITH (UPDLOCK);

Now, connect a second session. In the connection dialog, select Options».

In the connection options dialog, set Execution time-out to 5 seconds.

Select Connect.

Run the following query in this session:

SELECT
 max_date = MAX(fos.DateKey)
FROM dbo.FactOnlineSales AS fos;

This query will be blocked by the query with the open transaction. After five seconds, you should see the error:

Msg -2, Level 11, State 0, Line 0
Execution Timeout Expired. The timeout period elapsed prior to
completion of the operation or the server is not responding.

Now, back in the first session, run:

ROLLBACK

Find the Timeout in Query Store

To find timeouts in Query Store, you want to look for queries where execution_type in sys.query_store_runtime_stats is set to 3. Note that this isn’t only query timeouts: this status indicates ‘Client initiated aborted execution’. But in my testing I find that this includes query timeouts.

Here is a sample query against the Query Store DMVs looking for this status:

SELECT
 qst.query_sql_text,
 qrs.execution_type,
 qrs.execution_type_desc,
 qpx.query_plan_xml,
 qrs.count_executions,
 qrs.last_execution_time
FROM sys.query_store_query AS qsq
JOIN sys.query_store_plan AS qsp
 ON qsp.query_id = qsq.query_id
JOIN sys.query_store_query_text AS qst
 ON qst.query_text_id = qsq.query_text_id
OUTER APPLY
(
 SELECT
 query_plan_xml = TRY_CONVERT(XML, qsp.query_plan)
) AS qpx
JOIN sys.query_store_runtime_stats AS qrs
 ON qrs.plan_id = qsp.plan_id
WHERE qrs.execution_type = 3
ORDER BY
 qrs.last_execution_time DESC;
GO

Here’s a screenshot of the results I see in SSMS for this query:

I clicked the “cancel executing query” button for the queries on rows 2 and 3 of this result set. The query on row 1 was run twice with the execution time-out setting in SSMS set to 5 seconds when I had the table locked in another session.

This Isn’t Perfect, but It’s Worth Checking

As I mentioned, you probably don’t want to capture all queries in Query Store in a production database. When timeouts strike, there’s no guarantee that Query Store has a record of the query. (This is true for all monitoring systems, really: there’s a performance tradeoff in observing things incredibly closely!)

However, if you have Query Store enabled, I definitely think it’s worth checking it for information regarding timeouts, as it’s a pretty quick and painless process to do so. If it doesn’t have the timeout data you’re looking for, you can always start up a trace and look for those Attention events as usual.

There’s a Lot More Cool Things in Query Store

Query Store is an interesting thing. It doesn’t replace monitoring systems: if your database is inaccessible, you can’t get to Query Store to figure out what’s wrong. It doesn’t alert you as to downtime or availability, that’s not its purpose.

However, Query Store does replace a lot of the performance tuning aspects of monitoring systems. It certainly can allow you to go cheaper on a monitoring solution and primarily look for availability monitoring, while doing most of your tuning and troubleshooting with Query Store.

Index Design Guide Updated: Index Types, B+ Trees, and Row Locators, Oh My

help@KendraLittle.com (Kendra Little) — Wed, 23 Feb 2022 08:38:18 -0500

We’ve recently updated the SQL Server and Azure SQL index architecture and design guide. This article is an in-depth guide to indexing in databases using the SQL Server engine, including SQL Server, Azure SQL Database, Azure SQL Managed Instance, and Azure Synapse Analytics.

Our recent update adds a table to categorize the types of indexes discussed in the article, clarifies B-trees vs B+ trees, and describes how row locators (aka “secret columns”) are used in nonclustered indexes.

New Table Summarizing Types of Indexes

The index architecture and design guide now opens with a table describing the types indexes discussed in the article, grouped by their primary storage format: disk-based rowstore indexes, columnstore indexes, and memory-optimized indexes.

This is a small change, but I hope it proves helpful to folks who are newer to this topic and who may find themselves confused by different uses of the word ’nonclustered’, or who want a big-picture view of the most commonly used index types.

Clarifying B Trees vs B+ Trees

A customer submitted an issue asking us to clarify when the SQL Server engine uses B-trees vs B+ trees. Up to this point, the documentation for SQL Server and Azure SQL has largely used the general term “B-tree” without being more specific.

My colleague Mike Ray (Twitter) took on this project, which was a considerable amount of work: there are a lot of articles which mention B-trees. Most of these articles shouldn’t go into a deep discussion of types of B-trees, however. Mike identified these articles (for example, the CREATE INDEX (Transact-SQL) article) and added the following note:

SQL Server documentation uses the term B-tree generally in reference to indexes. In rowstore indexes, SQL Server implements a B+ tree. This does not apply to columnstore indexes or in-memory data stores. Review SQL Server Index Architecture and Design Guide for details.

Mike worked with Pedro Lopes (Twitter) to update the index architecture and design guide to clarify when B+ indexes are used (disk-based rowstore indexes, the deltastore for columnstore indexes) and when other variations of B-trees are used (a Bw-tree is used in In-Memory nonclustered indexes).

Together, these changes should not only help folks who are interested in B-trees, but also will help raise awareness of the index architecture and design guide itself as a great free resource for learning about indexing.

Describing Row Locators (aka “Secret Columns”) in Nonclustered Indexes

Another customer submitted an issue asking us to document how and why the SQL Server Engine adds clustered index keys to nonclustered indexes.

Based on this request, I worked with Pedro to add information to the nonclustered index architecture section of the design guide.

We went a little broader than the request, because if a disk-based rowstore table does not have a clustered index (in other words, if it is a heap), then a row locator is still added to nonclustered indexes. We added the following table, along with a table of specific examples.

And, importantly, we added the following sentence:

Clustered index key-based row locator columns in a nonclustered index can be used by the query optimizer, regardless of whether they were explicitly specified in the index definition.

Thanks to Pedro Lopes and Mike Ray

Thanks very much to my colleagues Mike Ray (Twitter) and Pedro Lopes (Twitter) for their efforts in helping make the SQL Server and Azure SQL index architecture and design guide even better.

Workers vs. Requests in Azure SQL Database

help@KendraLittle.com (Kendra Little) — Tue, 18 Jan 2022 17:36:30 -0500

We now explicitly define ‘requests’ and ‘workers’ in the Azure SQL Database documentation, and we’ve cleaned up multiple places where we used to equate the two terms. In this post, I share the history of the two terms when it comes to Azure SQL Database, why the two were ever equated, and why things like this are tricky to change.

What We Changed

The documentation for Azure SQL Database previously contained some statements which equate workers and requests. Notice that this heading equates workers and requests, but the paragraph talks only about worker limits:

Sessions and workers (requests)

The maximum numbers of sessions and workers are determined by the service tier and compute size. New requests are rejected when session or worker limits are reached, and clients receive an error message.

The documentation also has tables which explain resource limits by service tier and compute size. Those tables used to look like this:

An example table equating workers and requests

What Are Workers and Requests, Anyway?

We made changes to clarify that there is no limit on requests. Instead, there is a worker limit. We also added information to help folks troubleshoot situations where they approach or hit worker limits.

We now define sessions, requests, and workers:

A session represents a process connected to the database engine.

A request is the logical representation of a query or batch. A request is issued by a client connected to a session. Over time, multiple requests may be issued on the same session.

A worker thread, also known as a worker or thread, is a logical representation of an operating system thread. A request may have many workers when executed with a parallel query execution plan, or a single worker when executed with a serial (single threaded) execution plan. Workers are also required to support activities outside of requests: for example, a worker is required to process a login request as a session connects.

We’ve also cleaned up the tables: where tables used to read ‘workers (requests)’, they now simply read ‘workers’.
Finally, we added more troubleshooting steps for when the worker limit has been reached under Error 10928: Resource ID : 1. The request limit for the database is %d and has been reached.

Notice that this error message still uses the word ‘requests’: keep reading to learn why.

Why Were Workers and Requests Ever Equated for Azure SQL Database?

We added a note below that error message which explains:

The initial offering of Azure SQL Database supported only single threaded queries. At that time, the number of requests was always equivalent to the number of workers. Error message 10928 in Azure SQL Database contains the wording “The request limit for the database is N and has been reached” for backwards compatibility purposes. The limit reached is actually the number of workers.

Why Was This Tricky to Change?

When Azure SQL Database gained the ability to run parallel queries, requests and workers were no longer interchangeable at all: a single request may have a large amount of workers, depending on the query plan and level of parallelism.

But things like that error message are risky to change. One of the big requirements of providing a hosted database service is providing as much stability as possible while still innovating. Customers can, and do, sometimes code dependencies on the text of error messages (not just the number). Changing something even as simple as an error message can cause regressions.

The documentation attempted to strike a balance here: headings and tables were adjusted to say “workers” with the older term “(requests)” following in parentheses, while explanatory paragraphs talked only about the request limit.

Why We Updated the Docs Further

Feedback came in over the last few months that the documentation wasn’t striking the balance as intended: this was confusing. Based on that feedback, we made the changes above to clearly define the terms involved, explain and support folks hitting worker limits, and simplify the resource limits tables.

Thanks very much to Dimitri Furman for his help in working out these changes.

What I Learned Writing About How to Diagnose and Troubleshoot High CPU in Azure SQL Database

help@KendraLittle.com (Kendra Little) — Wed, 05 Jan 2022 10:56:52 -0500

Writing helps me learn. In my job as a Content Developer, this is more true than ever: there’s a fantastic group of folks, both in the Database Docs team and in the Microsoft Data Platform engineering team, who review and contribute to content.

I’ve just had the pleasure of publishing my first new article in the Microsoft Docs, Diagnose and troubleshoot high CPU on Azure SQL Database.

This article isn’t really “mine”: anyone in the community can create a Pull Request to suggest changes, or others at Microsoft may take it in a different direction. But I got to handle the outlining, drafting, and incorporation of suggested changes for the initial publication.

It was a ton of fun, and I learned a lot about Azure SQL Database in the process.

You Can Query Your vCore Count

Azure SQL Database lets you choose between two purchasing models: vCore and database transaction unit (DTU).

One benefit of the vCore model is that understanding and controlling the number of virtual cores allocated to your database is pretty straightforward: there’s a slider for the number of cores.

I did learn, however, that it can be worthwhile to query your virtual core count with Transact-SQL. That’s because:

You might have more vCores than you think. For databases using Gen4 hardware, the number of schedulers in sys.dm_os_schedulers may be double the number of vCores specified at database creation and shown in Azure portal.
If you are using the DTU purchasing model, you don’t get to specify the number of cores available.

We’ve got a simple query in the article which will return your current vCore count for a database.

You Can Switch Between Insights in the Azure Portal and Query Store

There’s a nice area in the Azure portal called Query Performance Insight. You can see how much CPU was used over time along with top queries that ran in that time period.

I like this view and find it pretty easy to navigate. I had a theory that the query information was coming from Query Store, but I wasn’t certain. I found that in fact, it is: you can take the query ids from the portal and plug them into the graphic Query Store tools in SQL Server Management Studio (SSMS) and dig into query execution plans and historic execution stats that way. I like that these can work together.

“In Flight” Execution Statistics Are Enabled by Default in Query Plans

This is something that’s been around for a little while, but I missed that this became available by default. There’s a DMV, sys.dm_exec_query_statistics_xml, which returns “in flight” statistics for execution plans for currently running queries.

It used to be that you could only see what SQL Server estimated for queries that are currently running. But sometimes estimates are way off: sometimes the optimizer thinks it’s only going to get one row back, but it turns out to be millions. Or the opposite might happen. In-flight statistics help you see how many rows have flowed through different parts of a query plan so far.

In-flight statistics rely on something called lightweight query execution statistics profiling. Earlier versions of lightweight profiling can be enabled on SQL Server 2014 or 2016.

The latest version of lightweight query execution statistics profiling is enabled by default on Azure SQL Database and in SQL Server 2019.

You Can Get Query Compilation Rates from Query Store

There are a few things in SQL Server which I’m used to collecting with perfmon counters. That can be handy if you’re used to it, but it lacks a level of detail: you get aggregate information about compilation overall, but you don’t know which queries were compiling or re-compiling. (And honestly, understanding what gets counted as a compile vs a recompile in perfmon counters is pretty confusing.)

In researching for the article, I found that you can query compilation rates from Query Store. This is great, because you can dig in and see which queries are being compiled most frequently during different time periods. You can query this with Transact-SQL.

Have Suggestions for the Article?

We’d love to read it! There’s two great ways to submit suggestions:

Select the ‘Edit’ button at the top right of the page and submit a suggestion via a Pull Request in GitHub. This can be done entirely in the browser, no major GitHub experience is required (although you’ll need a free account)
Head down to the ‘Feedback’ section at the very bottom of the article and select ‘This page’. This will take you to a screen where you can submit a GitHub issue detailing your thoughts. Again, you’ll need a free GitHub account for this, but no experience with GitHub is required.

Addressing the Rumors That I Have Been Removed from the MVP Program

help@KendraLittle.com (Kendra Little) — Thu, 04 Nov 2021 16:24:12 -0400

I’d like to openly and honestly answer the rumors that I have been removed from the Microsoft MVP program.

Firstly, It’s True

I’ve started a new job at Microsoft. As of Monday, I am now a proud member of the Content Development team which brings you documentation on all things SQL Server and Azure SQL. I will be working with the product team, other Content Developers, and community contributors while focusing on Azure SQL Database.

As a Microsoft employee, I’m no longer eligible to be a member of the MVP program. It’s been a privilege to be an MVP, but this was a job I couldn’t resist.

I Was Not Forced to Leave the United Kingdom

Some say I fled Boris Johnson’s hair.

While this is plausible, the truth is that Jeremiah and I have moved back to the United States to be closer to our families. We found that it was very hard traveling across oceans in a global pandemic and decided that the right thing to do was to move back from the UK. It made sense to shift our careers as part of this process.

Why I’m Excited About My New Job

I owe so much of my career to the Microsoft documentation on SQL Server. Hardly a workday has gone by over the last 20 years when I haven’t read, shared, discussed, or blogged about these docs. (And in my heart, I will always think of these docs as ‘Books Online’.)

As the SQL Server engine expands and as the Azure SQL offering evolves, the docs continue to help the community learn, grow, and share.

To me, this is hugely exciting and I’m so honored and thrilled to get to work in this space.

Want to Learn More About the SQL Server and Azure SQL Docs Team?

If you’d like to learn more about the team or how you can help out with the docs yourself, my teammate William Assaf is giving a session at the upcoming PASS Community Data Summit, Become a Contributor to Microsoft Docs.

The session is free and takes place on Thursday, November 11 at 2pm EST. There will be an interactive Q&A session with the team following William’s 30-minute talk.

A lot of the work I’ll be doing is collaborating with contributors and helping community members suggest changes and improvements to the documentation. Learning how to work in this process is simple. It also builds activity on your GitHub profile which can be a nice feature for your resume. I hope you can join us.

Gotchas with Deferred Name Resolution in SQL Server: Stored Procedures, Synonyms, Views, and Functions

help@KendraLittle.com (Kendra Little) — Mon, 18 Oct 2021 11:14:16 +0100

Have you ever tried to create an object in SQL Server, but it failed due to a missing table, column, or other dependency? If so, you’ve hit a case where SQL Server doesn’t offer ‘deferred name resolution’.

While these errors can be helpful when you’ve made a typo or accidentally used the wrong database, this can sometimes be a big hassle when you are…

Deploying database code to set up a partial environment
Deploying database code from version control to an empty database to ensure the code is valid

In this post, I walk through common scenarios and test whether deferred name resolution works or not.

What Is Deferred Name Resolution?

Deferred Name Resolution allows you to create objects in SQL Server without verifying that referenced objects (or referenced columns in objects) exist.

This is quite a confusing feature because it only works for some types of objects in specific scenarios. Every time I think I understand this feature, I am sure to discover a new gotcha.

Summary Table

I haven’t memorized everything in this post, and I don’t expect that you will, either. I’m creating this as a reference I can revisit when this topic comes up. Here’s a summary of when different object types do and do not get deferred name resolution, per my testing:

	Object doesn’t exist	Object exists with different columns	Object in an offline database	Object referenced via Linked Server
Stored Procedures	Y	N	Sometimes	Usually N
Synonyms	Y	N/A	Y	Y
Views	N	N	N	N
Scalar Functions	Y	N	Y	N
Inline Table Valued Functions	N	N	N	N
Multi-Statement Table Valued Functions	Y	N	Sometimes	Usually N

To dig into any area, use the table of contents above to jump to that section for demo code and a comment or two.

Working Around a Lack of Deferred Name Resolution

There is no special trick or trace flag which I know of to magically enable deferred name resolution where it doesn’t work. However, if you need to deploy code to an empty database or to a partial database environment and it is failing due to missing dependencies, there are a few techniques which you can leverage to help get around the errors.

Restore Databases as a “Base” for Deployments in Test Environments

Dependencies on objects are only checked when you run CREATE or ALTER code. These dependencies are not checked when you restore a database. It is possible to restore a database containing all sorts of invalid objects, in fact, such as views which reference tables that no longer exist.

This means that you can restore databases to support dependencies, whether they are cross-database or cross-server dependencies, and you can process the restore commands in any order you like.

Use Synonyms for Flexibility Across Environments

Although synonyms aren’t a simple fix for a lack of deferred name resolution, they’re incredibly helpful when it comes to cross-database and cross-server dependencies.

Synonyms provide a helpful level of abstraction which allows you to encapsulate any environment-specific information such as database names or linked server names in a single type of object (synonyms). One common design pattern is to:

Create synonyms for all object dependencies outside the current database
Deployments include a script which runs at the beginning of each deployment and checks that all synonyms are created properly for the target environment, and recreates them if needed

This has the benefits of making the definitions of stored procedures, views, and functions identical across databases: all objects refer to the same synonym. It also allows flexibility in environment configuration: perhaps in some environments all databases may be restored to the same SQL Server instance, while in other environments they are referred to multiple instances.

I recommend following this pattern for all cross-database dependencies, even in cases where SQL Server offers deferred name resolution. Being consistent at this helps developers and database administrators working with the database understand the configuration.

Dynamic SQL, If You Dare (I Do Not Dare in This Case)

Another option is to utilize Dynamic SQL and only create objects with external dependencies in some target environments.

I am a fan of Dynamic SQL for many use cases, but I’m not generally a huge fan of it for schema definitions. This makes it hard to understand which objects are in version control.

Don’t Use Linked Servers

This is easier said than done when it comes to legacy codebases, but avoid linked servers whenever possible.

When writing new code, always check to see if data can be managed entirely by application services and dependencies between databases can be kept to a minimum. Modern software trends generally advise on building applications with clear service boundaries and minimizing dependencies at the database level, so this usually should not be a radical request.

Stored Procedures Sometimes Get Deferred Name Resolution

Stored procedures have deferred name resolution for object creation in some cases. This means that you can often create a stored procedure referencing a table that doesn’t exist yet. When the stored procedure is executed for the first time, the SQL Server query processor will validate that dependencies are present.

✅ Schema and/or Object Does Not Exist

For example, SQL Server can successfully create this stored procedure even though my database does not have a schema named SchemaDoesNotExist.

CREATE OR ALTER PROCEDURE
 dbo.TestProcedure
AS
SELECT
 SomeColumn
FROM SchemaDoesNotExist.TableDoesNotExist;
GO

Similarly, if the schema exists but no table or view exists with the given name, the stored procedure can be created as well. This code succeeds:

CREATE OR ALTER PROCEDURE
 dbo.TestProcedure
AS
SELECT
 SomeColumn
FROM dbo.TableDoesNotExist;
GO

Stored procedures can reference temporary tables (including global temporary tables) which do not exist and be created successfully. This stored procedure can be created:

CREATE OR ALTER PROCEDURE
 dbo.TestProcedure
AS
SELECT
 SomeColumn
FROM #DoesNotExist;
GO

Deferred name resolution isn’t only for tables and views. Deferred name resolution also works when calling other stored procedures which don’t exist. This procedure is created successfully.

CREATE OR ALTER PROCEDURE
 dbo.TestProcedure
AS
EXECUTE dbo.ProcedureDoesNotExist;
GO

In this case, SQL Server prints an informational message:

The module 'TestProcedure' depends on the missing object 'dbo.ProcedureDoesNotExist'. The module will still be created; however, it cannot run successfully until the object exists.

We can also successfully create a procedure referencing a scalar or table valued function which does not exist. Here is an example for a table valued function:

CREATE OR ALTER PROCEDURE
 dbo.TestProcedure
AS
SELECT
 SomeColumn
FROM dbo.TableValuedFunctionDoesNotExist();
GO

❌ Table or View Exists but Doesn’t Have the Column Listed

Stored procedures cannot use deferred name resolution when they reference an object which does exist, but which does not contain a column listed. For example:

CREATE TABLE
 dbo.T1
(
 i integer IDENTITY
);
GO

CREATE OR ALTER PROCEDURE
 dbo.TestProcedure
AS
SELECT
 SomeColumn
FROM dbo.T1;
GO

This code throws an invalid column error and the stored procedure is not created.

Msg 207, Level 16, State 1, Procedure TestProcedure, Line 4 [Batch Start Line 2]
Invalid column name 'SomeColumn'.

❌ Stored Procedure in Another Database Which Is Offline

The rules above also apply to references to objects in other databases on the same SQL Server instance:

✅ You can create a stored procedure referencing objects which don’t exist in another database
✅ You can create a stored procedure referencing objects in a database which doesn’t exist
❌ You CANNOT create a stored procedure referencing a column which does not exist in a table which does exist in another database

However, if you attempt to reference an object in a database which is offline, that sometimes causes a problem.

To demonstrate this, first, create a database and set it offline:

IF DB_ID('PutMeOffline') IS NULL
BEGIN
 CREATE DATABASE PutMeOffline;
END;
GO

ALTER DATABASE
 PutMeOffline
SET OFFLINE;
GO

We get deferred name resolution if we reference a table in an offline database. This statement succeeds:

CREATE OR ALTER PROCEDURE
 dbo.TestProcedure
AS
SELECT
 *
FROM PutMeOffline.dbo.TableDoesNotExist;
GO

However, if we try to reference a stored procedure in an offline database, our create statement fails:

CREATE OR ALTER PROCEDURE
 dbo.TestProcedure
AS
EXECUTE PutMeOffline.dbo.ProcedureDoesNotExist;
GO

This returns the following error:

Msg 942, Level 14, State 4, Procedure TestProcedure, Line 3 [Batch Start Line 13]
Database 'PutMeOffline' cannot be opened because it is offline.

This error does not appear if I create a stored procedure referencing a table valued function in an offline database.

❌✅ Object Referenced via a Linked Server

Linked servers ruin everything.

Referencing an object via a four-part name breaks deferred name resolution, for the most part.

In this example, I reference a linked server which does not exist:

CREATE OR ALTER PROCEDURE
 dbo.TestProcedure
AS
SELECT
 SomeColumn
FROM LinkedServerDoesNotExist.DatabaseDoesNotExist.dbo.TableDoesNotExist;
GO

This fails with the error:

Msg 7202, Level 11, State 2, Procedure TestProcedure, Line 3 [Batch Start Line 0]
Could not find server 'LinkedServerDoesNotExist' in sys.servers. Verify that the correct server name was specified. If necessary, execute the stored procedure sp_addlinkedserver to add the server to sys.servers.

In this example, I use a valid linked server, but reference a database which does not exist on the remote server.

CREATE OR ALTER PROCEDURE
 dbo.TestProcedure
AS
SELECT
 SomeColumn
FROM [DERP\SEATTLE].DatabaseDoesNotExist.dbo.TableDoesNotExist;
GO

This fails with the error:

Msg 7314, Level 16, State 1, Procedure TestProcedure, Line 3 [Batch Start Line 0]
The OLE DB provider "SQLNCLI11" for linked server "DERP\SEATTLE" does not contain the table ""DatabaseDoesNotExist"."dbo"."TableDoesNotExist"". The table either does not exist or the current user does not have permissions on that table.

I get the same error if I reference a table that does not exist in a valid database across the linked server.

Everything here seems to have an exception. For whatever reason, I am able to get deferred name resolution when it comes to referencing a stored procedure across a linked server. This code succeeds:

CREATE OR ALTER PROCEDURE
 dbo.TestProcedure
AS
EXECUTE LinkedServerDoesNotExist.DatabaseDoesNotExist.dbo.ProcedureDoesNotExist;
GO

Synonyms Use Deferred Name Resolution, but Their Power Is Not Transitory

Synonyms are lightweight pointers in SQL Server. As the documentation mentions,

The base object need not exist at synonym create time. SQL Server checks for the existence of the base object at run time.

✅ Schema and/or Object Does Not Exist

This synonym can be created successfully:

CREATE SYNONYM dbo.TestSynonym
FOR dbo.TableDoesNotExist;
GO

✅ Object Referenced via a Linked Server

I can even create a synonym referencing a linked server which does not exist:

CREATE SYNONYM dbo.TestSynonym
FOR LinkedServerDoesNotExist.DatabaseDoesNotExist.dbo.TableDoesNotExist;
GO

✅ Object in Another Database Which Is Offline

Synonyms don’t seem to have any problems with offline databases, either. I tested this with the same offline database which I created earlier with this code:

IF DB_ID('PutMeOffline') IS NULL
BEGIN
 CREATE DATABASE PutMeOffline;
END;
GO

ALTER DATABASE
 PutMeOffline
SET OFFLINE;
GO

The following code to create a synonym referencing the offline database works.

🔥 Note: Synonyms don't support CREATE OR ALTER syntax. They do support DROP IF EXISTS.

DROP SYNONYM IF EXISTS dbo.TestSynonym;
GO

CREATE SYNONYM
 dbo.TestSynonym
FOR PutMeOffline.dbo.TableDoesNotExist;
GO

❌ Using a Synonym Doesn’t Fix a Lack of Deferred Name Resolution for Other Objects

I love synonyms and it’s terrific that all of these examples work, but don’t get too excited. Synonyms are not a one-stop workaround for a lack of deferred name resolution. ‘Run time’ can apparently mean creation of an object referencing the synonym.

For example, if I create a procedure that references TestSynonym as created above, the code still fails.

CREATE OR ALTER PROCEDURE
 dbo.TestProcedure
AS
SELECT
 SomeColumn
FROM dbo.TestSynonym;
GO

This results in the same error we saw above when referencing the table via a four-part name:

Msg 7202, Level 11, State 2, Procedure TestProcedure, Line 3 [Batch Start Line 8]
Could not find server 'LinkedServerDoesNotExist' in sys.servers. Verify that the correct server name was specified. If necessary, execute the stored procedure sp_addlinkedserver to add the server to sys.servers.

Similarly, synonyms aren’t a workaround that can instantly get you deferred name resolution for other objects which lack it, such as views.

Speaking of views…

Views Do Not Get Deferred Name Resolution

When you create a view in SQL Server, all dependencies need to be able to be resolved or the view will not be created successfully. This is true even if you do not create the view with SCHEMABINDING.

❌ Schema and/or Object Does Not Exist

This script attempts to create a view referencing a table which doesn’t exist in the current database. The script fails:

CREATE OR ALTER VIEW
 dbo.TestView
AS
SELECT
 *
FROM dbo.TableDoesNotExist;
GO

SQL Server returns Error 208 and does not create the view.

Msg 208, Level 16, State 1, Procedure TestView, Line 4 [Batch Start Line 0]
Invalid object name 'dbo.TableDoesNotExist'.

Similarly, if you try to create a view referencing a scalar function that does not exist, the script also fails. For example:

DROP TABLE IF EXISTS dbo.T1;
GO
CREATE TABLE dbo.T1 (i INT IDENTITY);
GO
CREATE OR ALTER VIEW dbo.TestView
AS
 SELECT dbo.SomeFunction(i)
 FROM dbo.T1;
GO

This code fails. SQL Server returns error 4121:

Msg 4121, Level 16, State 1, Procedure TestView, Line 3 [Batch Start Line 5]
Cannot find either column "dbo" or the user-defined function or aggregate "dbo.SomeFunction", or the name is ambiguous.

❌ Table or View Exists but Doesn’t Have the Column Listed

SQL Server must resolve not only the objects referenced, but also the columns referenced when you create a view. For example, this code attempts to use a column which doesn’t exist in the referenced table:

DROP TABLE IF EXISTS dbo.T1;
GO
CREATE TABLE dbo.T1 (i INT IDENTITY);
GO
CREATE OR ALTER VIEW dbo.TestView
AS
 SELECT ColumnDoesNotExist
 FROM dbo.T1;
GO

This code fails with error 207 and the view is not created.

Msg 207, Level 16, State 1, Procedure TestView, Line 3 [Batch Start Line 4]
Invalid column name 'ColumnDoesNotExist'.

❌ Object in Another Database Which Is Offline

You cannot create a view referencing an object in an offline database, as attempted in this code:

IF DB_ID('PutMeOffline') IS NULL
 CREATE DATABASE PutMeOffline;
GO
ALTER DATABASE PutMeOffline SET OFFLINE
GO
CREATE OR ALTER VIEW dbo.TestView
AS
 SELECT *
 FROM PutMeOffline.dbo.T1;
GO

This fails with error 942:

Msg 942, Level 14, State 4, Procedure TestView, Line 3 [Batch Start Line 5]
Database 'PutMeOffline' cannot be opened because it is offline.

❌ Object Referenced via a Linked Server (Distributed View)

The above rules hold true for tables in other databases referenced using three-part naming, as well as tables referenced across linked servers using four-part naming. SQL Server must validate that the objects and columns referenced exist, or it cannot create the view.

Here is an example for a linked server:

CREATE OR ALTER VIEW dbo.TestView
AS
 SELECT *
 FROM SomeLinkedServer.dbo.TableDoesNotExist;
GO

This fails with error 208:

Msg 208, Level 16, State 1, Procedure TestView, Line 4 [Batch Start Line 0]
Invalid object name 'SomeLinkedServer.dbo.TableDoesNotExist'.

❌ Synonym References Object Which Doesn’t Exist

As mentioned above, synonyms can be created referencing objects which don’t exist, but this isn’t a simple workaround for objects which lack deferred name resolution. This code successfully creates a synonym, dbo.TestSynonym, which references a table which doesn’t exist. However, the code to create the view which references the synonym fails:

DROP SYNONYM IF EXISTS dbo.TestSynonym;
GO
CREATE SYNONYM dbo.TestSynonym
FOR dbo.TableDoesNotExist;
GO
CREATE OR ALTER VIEW dbo.TestView
AS
 SELECT*
 FROM dbo.TestSynonym;
GO

SQL Server returns error 5313 in this case:

Msg 5313, Level 16, State 1, Procedure TestView, Line 4 [Batch Start Line 5]
Synonym 'dbo.TestSynonym' refers to an invalid object.

🔄 How Circular Views Come into Existence

It is possible to have circular references between views in SQL Server. In this case, 2 or more views are mutually dependent upon one another. This typically happens by a process like the following:

View dbo.V1 is created referencing only tables in the database
View dbo.V2 is created with a reference to dbo.V1
View dbo.V1 is edited and a dependency to dbo.V2 is added

SQL Server allows this progression of changes to be deployed, but this results in a situation where views dbo.V1 and dbo.V2 cannot be successfully created in an empty database using their full definitions.

Function Behavior Varies by Type

I’m looking at scalar, inline (single statement) table valued functions, and multi-statement table valued functions separately because some get deferred name resolution and some don’t.

Scalar Functions

✅ Schema and/or Object Does Not Exist

Scalar functions can get deferred name resolution. The following code referencing a table which does not exist succeeds:

CREATE OR ALTER FUNCTION
 dbo.TestScalarFunction()
RETURNS varchar(128)
AS
BEGIN
 DECLARE
 @i integer;

 SELECT
 @i = 2
 FROM dbo.TableDoesNotExist;

 RETURN 1;
END;
GO

You might wonder if a call to this function works, since 1 is returned no matter what happens within the function. Let’s test that out with the following code:

SELECT dbo.TestScalarFunction();
GO

This fails with error 208:

Msg 208, Level 16, State 1, Line 11
Invalid object name 'dbo.TableDoesNotExist'.

❌ Table or View Exists but Doesn’t Have the Column Listed

Just like with stored procedures, if you create a scalar function referencing a table or view which exists, SQL Server will check the columns you specify and require that they are valid. This code to create dbo.TestScalarFunction fails:

DROP TABLE IF EXISTS dbo.T1;
GO
CREATE TABLE dbo.T1 (i INT IDENTITY);
GO
CREATE OR ALTER FUNCTION dbo.TestScalarFunction()
RETURNS VARCHAR(128)
AS
BEGIN
 DECLARE @i INT;
 SELECT @i=DoesNotExist FROM dbo.T1;
 RETURN 1;
END
GO

SQL Server returns error 207 and does not create the function.

Msg 207, Level 16, State 1, Procedure TestScalarFunction, Line 6 [Batch Start Line 4]
Invalid column name 'DoesNotExist'.

✅ Object in Another Database Which Is Offline

Like stored procedures scalar functions which reference tables in offline databases can be created successfully. This code succeeds:

IF DB_ID('PutMeOffline') IS NULL
 CREATE DATABASE PutMeOffline;
GO
ALTER DATABASE PutMeOffline SET OFFLINE
GO
CREATE OR ALTER FUNCTION dbo.TestScalarFunction()
RETURNS VARCHAR(128)
AS
BEGIN
 DECLARE @i INT;
 SELECT @i=2 FROM PutMeOffline.dbo.TableDoesNotExist;
 RETURN 1;
END
GO

Stored procedures have a gotcha where they fail if they reference another stored procedure in an offline database. I don’t believe that scalar functions can call stored procedures 🤔, so there is no issue here.

I did test a scalar function which calls a scalar function in an offline database. This code succeeds:

IF DB_ID('PutMeOffline') IS NULL
 CREATE DATABASE PutMeOffline;
GO
ALTER DATABASE PutMeOffline SET OFFLINE
GO
CREATE OR ALTER FUNCTION dbo.TestScalarFunction()
RETURNS VARCHAR(128)
AS
BEGIN
 DECLARE @i INT;
 SELECT @i=dbo.SomeFunction(2) FROM PutMeOffline.dbo.TableDoesNotExist;
 RETURN 1;
END
GO

❌ Object Referenced via a Linked Server

Creating a scalar function referencing an object across a linked server seems like a recipe for extreme pain, but SQL Server does allow this.

However, to create the function SQL Server will need to cross the linked server and validate that the object, and any columns referenced in it, exist.

CREATE OR ALTER FUNCTION dbo.TestScalarFunction()
RETURNS VARCHAR(128)
AS
BEGIN
 DECLARE @i INT;
 SELECT @i=2 FROM LinkedServerDoesNotExist.SomeDatabase.dbo.TableDoesNotExist;
 RETURN 1;
END
GO

Inline Table Valued Functions (TVFs)

Unlike scalar functions, inline TVFs do NOT get deferred name resolution.

❌ Schema and/or Object Does Not Exist

CREATE OR ALTER FUNCTION dbo.TestInlineTableValuedFunction()
RETURNS TABLE
AS
 RETURN ( SELECT * FROM dbo.TableDoesNotExist );
GO

This fails with the error:

Msg 208, Level 16, State 1, Procedure TestInlineTableValuedFunction, Line 4 [Batch Start Line 0]
Invalid object name 'dbo.TableDoesNotExist'.

❌ Table or View Exists but Doesn’t Have the Column Listed

If you reference a table which exists in an inline TVF, you can only reference columns which exist. Here is our demo code:

DROP TABLE IF EXISTS dbo.T1;
GO
CREATE TABLE dbo.T1 (i INT IDENTITY);
GO
CREATE OR ALTER FUNCTION dbo.TestInlineTableValuedFunction()
RETURNS TABLE
AS
 RETURN ( SELECT DoesNotExist FROM dbo.T1 );
GO

This fails with error 207:

Msg 207, Level 16, State 1, Procedure TestInlineTableValuedFunction, Line 4 [Batch Start Line 4]
Invalid column name 'DoesNotExist'.

❌ Object in Another Database Which Is Offline

Inline TVFs cannot reference objects in offline databases. Here is our example code:

IF DB_ID('PutMeOffline') IS NULL
 CREATE DATABASE PutMeOffline;
GO
ALTER DATABASE PutMeOffline SET OFFLINE
GO
CREATE OR ALTER FUNCTION dbo.TestInlineTableValuedFunction()
RETURNS TABLE
AS
 RETURN ( SELECT * FROM PutMeOffline.dbo.SomeTable );
GO

This fails with the error:

Msg 942, Level 14, State 4, Procedure TestInlineTableValuedFunction, Line 4 [Batch Start Line 5]
Database 'PutMeOffline' cannot be opened because it is offline.

❌ Object Referenced via a Linked Server

If you reference a linked server in your inline TVF definition, SQL Server will need to successfully access the linked server and validate that the object and any columns referenced exists. Here is our demo code:

CREATE OR ALTER FUNCTION dbo.TestInlineTableValuedFunction()
RETURNS TABLE
AS
 RETURN ( SELECT * FROM LinkedServerDoesNotExist.SomeDatabase.dbo.TableDoesNotExist );
GO

This code fails with error 7202:

Msg 7202, Level 11, State 2, Procedure TestInlineTableValuedFunction, Line 4 [Batch Start Line 0]
Could not find server 'LinkedServerDoesNotExist' in sys.servers. Verify that the correct server name was specified. If necessary, execute the stored procedure sp_addlinkedserver to add the server to sys.servers.

Multi-Statement Table Valued Functions (TVFs)

Multi-statement table valued functions can get deferred name resolution sometimes. In my testing, I found the behavior of multi-statement TVFs to match the behavior of stored procedures regarding deferred name resolution.

✅ Schema and/or Object Does Not Exist

This create statement for a multi-statement table valued function succeeds, although it references a table which does not exist:

CREATE OR ALTER FUNCTION
 dbo.TestMultiStatementTableValuedFunction()
RETURNS @doggo TABLE
(
 doggo varchar(128)
)
AS
BEGIN
 INSERT @doggo
 (
 doggo
 )
 SELECT
 doggo
 FROM dbo.TableDoesNotExist;

 RETURN;
END;
GO

❌ Table or View Exists but Doesn’t Have the Column Listed

Consistent with what we’ve seen in other examples, if an object exists then any columns listed for the object must exist as well. Here is our example code to test this in a multi-statement table valued function:

DROP TABLE IF EXISTS dbo.T1;
GO

CREATE TABLE
 dbo.T1
(
 i integer IDENTITY
);
GO

CREATE OR ALTER FUNCTION
 dbo.TestMultiStatementTableValuedFunction()
RETURNS @doggo TABLE
(
 doggo varchar(128)
)
AS
BEGIN
 INSERT @doggo
 (
 doggo
 )
 SELECT
 DoesNotExist
 FROM dbo.T1;

 RETURN;
END;
GO

This fails with error 207:

Msg 207, Level 16, State 1, Procedure TestMultiStatementTableValuedFunction, Line 4 [Batch Start Line 4]
Invalid column name 'DoesNotExist'.

❌ Object in Another Database Which Is Offline

Like stored procedures, I found that multi-statement TVFs can reference tables and views in offline databases. This code executes successfully:

IF DB_ID('PutMeOffline') IS NULL
BEGIN
 CREATE DATABASE PutMeOffline;
END;
GO

ALTER DATABASE
 PutMeOffline
SET OFFLINE;
GO

CREATE OR ALTER FUNCTION
 dbo.TestMultiStatementTableValuedFunction()
RETURNS @doggo TABLE
(
 doggo varchar(128)
)
AS
BEGIN
 INSERT @doggo
 (
 doggo
 )
 SELECT
 *
 FROM PutMeOffline.dbo.T1;

 RETURN;
END;
GO

However, if I add a statement which calls a stored procedure in the offline database, the code fails:

IF DB_ID('PutMeOffline') IS NULL
BEGIN
 CREATE DATABASE PutMeOffline;
END;
GO

ALTER DATABASE
 PutMeOffline
SET OFFLINE;
GO

CREATE OR ALTER FUNCTION
 dbo.TestMultiStatementTableValuedFunction()
RETURNS @doggo TABLE
(
 doggo varchar(128)
)
AS
BEGIN
 EXECUTE PutMeOffline.dbo.ProcedureDoesNotExist;

 INSERT @doggo
 (
 doggo
 )
 SELECT
 *
 FROM PutMeOffline.dbo.T1;

 RETURN;
END;
GO

This fails with error 942:

Msg 942, Level 14, State 4, Procedure TestMultiStatementTableValuedFunction, Line 4 [Batch Start Line 19]
Database 'PutMeOffline' cannot be opened because it is offline.

❌✅ Object Referenced via a Linked Server

Finally, if you attempt to create a multi-statement TVF which references a table or view via a linked server, as with our other test cases SQL Server must access the linked server and validate that the object and any referenced columns exist.

CREATE OR ALTER FUNCTION
 dbo.TestMultiStatementTableValuedFunction()
RETURNS @doggo TABLE
(
 doggo varchar(128)
)
AS
BEGIN
 INSERT @doggo
 (
 doggo
 )
 SELECT
 doggo
 FROM LinkedServerDoesNotExist.SomeDatabase.dbo.TableDoesNotExist;

 RETURN;
END;
GO

This fails with error 7202:

Msg 7202, Level 11, State 2, Procedure TestMultiStatementTableValuedFunction, Line 4 [Batch Start Line 0]
Could not find server 'LinkedServerDoesNotExist' in sys.servers. Verify that the correct server name was specified. If necessary, execute the stored procedure sp_addlinkedserver to add the server to sys.servers.

However, as with stored procedures, multi-statement TVFs might execute a procedure across a linked server. Oddly enough, this code succeeds:

CREATE OR ALTER FUNCTION dbo.TestMultiStatementTableValuedFunction()
RETURNS @doggo TABLE(doggo VARCHAR(128))
AS BEGIN
 EXEC LinkedServerDoesNotExist.SomeDatabase.dbo.Something
 RETURN;
END;
GO

Learn Azure SQL with One Free Learning Path

help@KendraLittle.com (Kendra Little) — Thu, 14 Oct 2021 09:03:05 +0100

Have you been meaning to learn about SQL Server in the Azure cloud, but never seem to get around to it? It’s easy to be overwhelmed and not know where to begin.

I recommend you start with the Azure SQL Fundamentals Learning Path.

I stepped through this training this week and WOW, this is an incredibly high quality free course, complete with hands-on exercises in an online sandbox. I wish I’d done this sooner!

Top-Notch Quality

I say this is a high quality training because the course explains both general concepts and is also full of specific, useful, current details about Azure SQL.

Having written training modules myself, I know how difficult this is to maintain, especially given the rapid rate of evolution of cloud technologies. This Learning Path is clearly frequently updated and kept current by a team with very deep knowledge of Azure SQL.

Why Learn Azure SQL?

It’s increasingly useful for database administrators and developers to understand PaaS (Platform as a Service) and IaaS (Infrastructure as a Service) cloud offerings and how these compare to installing and managing databases on-prem.

Even if your current job doesn’t work with databases in the cloud, learning the fundamentals of Azure SQL is likely going to help you out when it comes to seeking a promotion or landing a new job.

Note that I say fundamentals here. Not everyone needs to become an expert, but understanding the fundamentals is starting to become table stakes for database jobs.

What Does ‘Azure SQL’ Mean?

‘Azure SQL’ is a term that encompasses three types of offerings:

Azure SQL Database: a PaaS offering (database as a service)
Azure SQL Managed Instance: a PaaS offering (a SQL Server instance as a service)
SQL Server running on Azure Virtual Machines: an IaaS offering

The course largely focuses on the details of Azure SQL Database and Azure SQL Managed Instances. This makes sense as SQL Server running on Azure Virtual Machines is broadly similar to running SQL Server on Virtual Machines in any environment.

Sandbox examples typically leverage Azure SQL Database in the course, which I suspect is because these databases spin up in a matter of minutes, while Azure SQL Managed Instances can still take up to six hours to spin up. I found this to be sufficient for learning fundamentals. The course provides a lot of details and screenshots about Azure SQL Managed Instances which I found very helpful.

How to Tackle the Learning Path

I recommend following a strategy related to your level of experience and knowledge about SQL Server.

SQL Server Beginners

If you are more of a generalist or are newer to SQL Server, this course can still be great for you but make sure not to get distracted by the details. There are tons of facts in this course which you could dig into and research for hours.

I recommend timeboxing your learning and focusing on the big picture: you’ll get a ton of value out of the course. You can always come back and dig into specific modules more afterward.

SQL Server Experts

If you have a lot of experience with SQL Server, I suspect that you’ll find the course answers a lot of the questions you might have about performance, scalability, high availability, and disaster recovery. I encourage you to:

Dig into the command line examples, even if you’re normally one to use a graphic portal
Go off-road and explore more around the given examples

Why the ‘Sandbox’ Is Particularly Cool

An exercise after I have launched the sandbox

I really like how the hands-on examples are implemented in these Learning Paths. ‘Sandboxes’ are integrated into the course itself. When you launch the sandbox it connects to your Azure Account, but it does something really cool: it creates a special Concierge folder for you which utilizes a Concierge subscription. Your Sandbox runs real resources in Azure, it is not a simulation.

I like this for multiple reasons. First, it was easy for me to get working. Second, it’s a simple way for users to not have to worry about costs. Finally, it segments off your learning away from any “normal” Azure usage you might have. The Sandbox environments are regularly cleaned up, so you don’t need to worry about forgetting to run a clean-up script if something comes up and you get pulled away from an exercise before you complete it.

Links to Get Going

New Tricks I Learned from the Azure Data Studio Code Editor Tutorial

help@KendraLittle.com (Kendra Little) — Thu, 07 Oct 2021 15:00:03 +0100

Today I walked through the Use Azure Data Studio to connect and query Azure SQL database Quickstart. This Quickstart is solid and is great for someone new to Azure Data Studio.

At the end of the Quickstart it suggested I try the Tutorial: Use the Transact-SQL editor to create database objects - Azure Data Studio. The tutorial taught me a couple of things that I’ve not noticed about Azure Data Studio, even though I’ve used it for a couple of years.

Built-In Snippets Are Quite Useful

I had no idea that Azure Data Studio had built in snippet functionality. And I admit that as soon as I skimmed the tutorial and saw there were snippets, I tried to use them the wrong way. I assumed that I’d open the Command Palette to insert a snippet. Nope!

To access snippets, simply type sql into the query editor itself. This will initialize a drop-down list which you can arrow through:

You Can Create Your Own Snippets

There are built-in snippets for lots of things: data definition changes like creating indexes or adding columns, data modification tasks like inserting or deleting rows, or management tasks like listing databases or finding space used.

But you might want to create your own snippets, too.

‘Peek Definition’ Functionality Exists, Too

I also hadn’t realized that Azure Data Studio supports Peek Definition functionality natively. The example given in a tutorial of peeking at a table’s definition when writing an insert statement is right on.

Bonus: Dragging the Columns Folder into a Query Edit Works Just Like in SSMS

Stepping through the Peek Definition example made me wonder if one of my favorite lazy hacks from SQL Server Management Studio works in Azure Data Studio. Sure enough, it does! You can write out the list of columns in a table in your query editor by dragging the column folder for the table into the query editor window:

Azure Data Studio Isn’t Just for Azure SQL

Don’t let the name fool ya. Azure Data Studio works for both Azure SQL and on-prem SQL Server installations.

Trying Out the Azure SQL Database Quickstart

help@KendraLittle.com (Kendra Little) — Mon, 04 Oct 2021 11:19:52 +0100

I’m working through a series of tutorials and learning paths for Azure SQL and sharing bite-sized reviews.

In this first walkthrough, I’m stepping through the Create a Single Database tutorial.

A Nice Feature: Choose If You Wish to Work in the Graphic Portal or in a Command Line

It’s really nice that the Quickstart offers multiple options to learn.

You can get instructions to walk through steps in the portal if you prefer, or you get three different options for working in a command line. Switching between these is as simple as clicking on tab headings in the Quickstart:

It’s convenient that you can run the Azure Cloud Shell in the same browser page as the documentation to try out the example. It’s nice to not need to open a separate terminal and toggle between windows.

What’s the Difference Between the Azure CLI, Azure CLI (sql up), and PowerShell Examples?

I was immediately a bit puzzled as to why I have three command line options. This is a lot of choices!

Here’s how the three command line options compare and why you might choose one over another:

Option	Description	Why you would use it	Type of Azure SQL Database created in Quickstart
Azure CLI	This example utilizes a variety of commands that are part of the Azure CLI (command line interface). This example is written for the Bash shell.	You want more granular control over the database creation process. You are comfortable with or wish to learn Bash syntax.	Serverless, Gen 5, 2 vCores
Azure CLI (sql up)	This example utilizes the Azure CLI and the example is written in the Bash shell. This example is much shorter, as it leverages the az sql up command, which can create the resource group, logical server, and database in a single command.	You want a single Bash command to quickly create an Azure SQL Database.	Provisioned, Gen 5, 2 vCores
PowerShell	This example utilizes the Azure CLI but is written with PowerShell cmdlets and utilizes PowerShell syntax (instead of Bash commands and bash syntax).	You want more granular control over the database creation process. You are comfortable with or wish to learn PowerShell.	Serverless, Gen 5, 2 vCores

Note that one line examples of the Azure CLI will work in any shell.

The PowerShell Example Worked Perfectly

I’ve created Azure SQL Databases in the Portal quite a few times in the past. I have less experience using the command line options to create and manage Azure SQL Databases, so I wanted to try those out and learn more.

First I stepped through the PowerShell example using the in-browser Azure Cloud Shell. That worked very well.

At one point I needed to work on something else in the middle of trying the example. I found I needed to restart the browser shell, and it cleared out the parameters I was using. I simply restarted the example from the beginning and it all worked perfectly a second time through, even though I’d stopped mid-way.

The “sql up” Example Threw an Error

Next I stepped through the Azure CLI (sql up) example. In the PowerShell example you could simply copy and past all the commands without modifying them, with the exception of needing to specify your own IP address for a firewall exception. That didn’t work this time. I ran the command as given:

az sql up \
 --resource-group $resourceGroupName \
 --location $location \
 --server-name $serverName \
 --database-name mySampleDatabase \
 --admin-user $adminlogin \
 --admin-password $password

This resulted in an error:

argument --resource-group/-g: expected one argument

I looked at what I’d run and realized that I hadn’t ever initialized the variables $resourceGroupName, $location, $serverName, $adminlogin, and $password. I needed to either add a block of text assigning those values, remove the variables altogether (sql up will create defaults), or use literal values.

I changed my command to this:

az sql up \
 --resource-group myResourceGroup \
 --location eastus \
 --server-name server-$RANDOM \
 --database-name mySampleDatabase \
 --admin-user azureuser \
 --admin-password Azure1234567!

This worked, and created a resource group, logical server, and Azure SQL Database all in one step!

This was pretty fun to see this happen all in one command.

Highlights and Lowlights

There’s a lot to like about the Quickstart experience with Azure:

One thing I’ve noticed over the years is that Microsoft has made it incredibly easy to configure initial firewall rules when setting up an Azure SQL Database. Configuring the firewall is part of the examples. If you are using the Azure Portal, SQL Server Management Studio, or Azure Data Studio to query an Azure SQL Database, they will attempt to help you configure firewall rules if needed.
Another thing I noticed is that the Quickstarts create a resource group at the beginning of the example. At the end of the example, they show you how to clean up everything you created by deleting the resource group. That’s a nice, simple way to help the user not need to manually delete items or risk leaving things around.

If there’s a lowlight, it’s simply that it can take a bit of time for things to spin up.

This is totally reasonable: when you step through the Quickstart, you are creating real resources in Azure. There’s a fair amount going on in the background.

However, you might be waiting for a minute or two at different points along the way. One nice thing about having the Azure Cloud Shell in the browser with the documentation is that you can read the documentation on the same page while you wait, and just keep an eye out to see when your command completes.

Tutorial: Create a Disposable SQL Server Database Container in an Azure DevOps Pipeline With Spawn

help@KendraLittle.com (Kendra Little) — Tue, 13 Jul 2021 11:13:31 +0100

This post is part of TSQLTuesday #140, “What have you been up to with containers?”

Disposable databases are indispensable for software developers, both for initial development work and in automation pipelines. Containers pair exceptionally well with temporary databases and provide a lightweight mechanism to spin up an environment on demand.

In this post, I share why we need disposable databases, use cases for disposable databases in automated workflows, and a tutorial to create an Azure DevOps pipeline to spin up an AdventureWorks2019 sample data container on demand using Spawn. Spawn has a free tier which you can try out anytime.

Why We Need Disposable Databases

There are two main reasons why we need disposable databases for software development:

Testing database changes requires a realistic dataset. Schema isn’t enough! Will that unique index create successfully in production? Can you add a non-nullable column to that table without a default value? How will this query perform? We need realistic datasets for these scenarios and more.
Databases generally don’t have good ways to “reset” after you make changes. I say “good” here because some techniques do exist. SQL Server, for example, has a database snapshot feature. One can create a database snapshot at a point in time, make some changes, and later revert to that snapshot. But this feature doesn’t scale for software development. Database snapshots aren’t portable: they are tightly coupled with the source database. Database snapshots are read-only for a point in time: they are not writeable. The database snapshot feature is also cumbersome to implement and wasn’t designed with automation in mind.

Without disposable databases, it is much harder to do quality testing of changes. It’s also much harder to validate deployments when reordering changes in release pipelines.

A Containerized Service for Ephemeral Database Containers

Redgate’s Foundry team created Spawn to explore how a containerized service can solve the problems above. The Spawn cloud service lets users:

Create data images. Your image can be empty, you can create it from scripts, or you can create it from database backups.
Create data containers from images. These containers can be quickly created, used, and then removed.

These are just the basics. There is a lot more functionality available, including resetting containers and graduating containers to new images.

Two Use Cases for Disposable Databases in Pipelines

Continuous Integration Builds

When your repository contains database code, it’s essential to ensure the code is valid, just like any other code. Disposable databases allow you to automate code validation for your database code whenever someone commits or merges changes into a code branch. Continuous Integration can call a pipeline or action which will:

Create a data container including the database
Deploy changes from the repo to the database
Fail the action or pipeline if the code doesn’t deploy successfully
Clean up the data container at the end of the job

Pull Requests

One example of using a disposable database in an automation pipeline is in a Pull Request (PR) workflow when using Git. In this workflow, a developer will:

Create a local Git branch
Make database changes locally to their environment (maybe a disposable database which they spin up just for the branch)
Save and commit their changes to Git locally using the team’s preferred tooling for the project
Push their branch up to the central repo
Create a Pull Request on the central repo to merge the code into a shared branch (commonly “main”)

There is often an option for automation to kick off at this point to validate the code in your branch. Effective automation will save work for the folks reviewing your Pull Request while enabling them to do high quality review of your changes. If you have a database cloning solution, this automation can:

Create a data container including the database
Deploy changes from the repo to the database

Optionally, you can keep the database around for the reviewers to inspect as part of the PR review. Many teams prefer to build an entire environment (database, application components, etc.) for use in PR reviews.

Prerequisites to Follow This Tutorial

All of the accounts, services, and tools used in this tutorial offer free tiers. To follow along, you need:

A GitHub account to connect to Spawn.
An Azure DevOps project - the free tier works fine. The project repo can be empty (no database code is required).
To install the Spawn command line on a computer. This has a few simple steps: log in to the Spawn web app with GitHub, download the Spawn command line executable to a local folder, add the folder to your path, and then run spawnctl auth in a terminal.

Note: We won’t deploy database code to the data container in this tutorial itself, but we’ll set the foundation to do that in a future tutorial.

Create an Azure DevOps Pipeline

In this example, I am working with a newly created Azure DevOps project. I’ve initialized the default Git repo with a readme, but I haven’t added any code.

First, click Pipelines in the left menu bar, then click New pipeline at the top right.

At the Where is your code? prompt, click Azure Repos Git. This is a YAML pipeline. (You could also do this process with code stored in Bitbucket Cloud or GitHub.)

At the Select a repository prompt, click on the name of the repo.

Troubleshooting: If you do not see a repo name here and this is a new Azure DevOps project, click on Repos in the left bar. If it says “[YourRepoName] is empty. Add some code!” at the top, one way to get it working fast is to scroll to the bottom of the page and click Initialize with the “Add a README” option ticked.

At the Configure your pipeline prompt, select Starter pipeline.

This should bring you to the Review your pipeline YAML screen, with simple starter code in a YAML pipeline.

Add YAML to the Pipeline to Create a Spawn Data Container

Our next step is to replace that starter pipeline code with the following YAML. This YAML will…

Use an Azure-hosted agent running ubuntu-latest
Download and install the Spawn command line (spawnctl) to the agent
Create a data container based on the image name specified in a variable named $(DATA_IMAGE_NAME) with a lifetime of 20 minutes
Authenticate to Spawn by using an access token stored in a variable named $(SPAWNCTL_ACCESS_TOKEN)

trigger:
- main

pool:
 vmImage: ubuntu-latest

# variable: $(SPAWNCTL_ACCESS_TOKEN) - https://docs.spawn.cc/commands/spawnctl-accesstoken-get
# Variable: $(DATA_IMAGE_NAME) - to test quickly use a public image https://docs.spawn.cc/other/public-data-images 

steps:
 - script: |
 echo "Downloading and installing spawnctl..."
 curl -sL https://run.spawn.cc/install | sh
 displayName: Install spawnctl
 - script: |
 set -e
 export PATH=$PATH:$HOME/.spawnctl/bin
 dataContainer=$(spawnctl create data-container --image $(DATA_IMAGE_NAME) --lifetime 20m -q )
 echo "##vso[task.setvariable variable=dataContainerName]$dataContainer"
 dataContainerJson=$(spawnctl get data-container $dataContainer -o json)
 port=$(echo $dataContainerJson | jq -r .port)
 host=$(echo $dataContainerJson | jq -r .host)
 user=$(echo $dataContainerJson | jq -r .user)
 password=$(echo $dataContainerJson | jq -r .password)
 displayName: spawn a database
 env:
 SPAWNCTL_ACCESS_TOKEN: $(SPAWNCTL_ACCESS_TOKEN)

Add Variables to Your Pipeline

On the Review your pipeline YAML screen, click Variables in the top right corner.

In the Variables pane, click New Variable

Add a new variable:

Set the variable name to DATA_IMAGE_NAME
Set the value to adventure-works:2019, or use the public image of your choice.

Click OK at the bottom of the pane.

In the Variables pane, click the plus sign to add a second variable.

To get the value for this variable, you need to:

Install the Spawn command line on a computer, if you have not done so already. (This should be fast: the page guides you to log into the Spawn web app with GitHub, download the Spawn command line executable, save it to a folder, add the folder to your path, and then run spawnctl auth in a terminal.)
Run the following command to create an access token:

spawnctl create access-token --purpose "Spawn access token for pipeline test"

Copy the token from the terminal to your clipboard.

Now, back in the Azure DevOps variables pane, add a second variable:

Set the variable name to SPAWNCTL_ACCESS_TOKEN
Past the value of your access token from your clipboard
Tick off the “keep this value secret” box to protect your access token

Click OK at the bottom of the pane.

Back in the main Variables pane, click Save at the bottom of the screen.

Save and Run Your Pipeline

As a quick recap, so far we have:

Created a starter YAML pipeline
Replaced the code with YAML to work with Spawn
Added two variables to the pipeline and saved them

Now we need to save the pipeline itself. On the Review your pipeline YAML screen, click Save and run.

In the Save and run pane which appears, click Save and run in the bottom right corner.

You will now be taken to a screen where you can watch your Pipeline job be queued and then run. If all has gone well, you should see your job succeed in a relatively short amount of time.

Check Out the Data Container Created by the Pipeline

Our pipeline asked Spawn to create a copy of the AdventureWorks sample database with a lifetime of 20 minutes. We can connect to that database in SQL Server Management Studio or Azure Data Studio if we’d like to check it out.

We let Spawn name the database container in this case.

Open a terminal, and run the following command to get a list of all your data containers.

spawnctl get data-containers

This will return a list of your containers with their metadata.

Next, plug the name of your data container into the following command.

spawnctl get data-container adventure-works-ebnmatep -o json

This will output connection information for your database.

Note from the output:

The Data Source connection string, including the port
The username is sa
A random password has been generated

You can use this information to connect to the SQL Server instance using the IDE of your choice.

If you enjoy using Azure Data Studio, the Spawn extension for Azure Data Studio makes connecting to your Spawn instances simple.

Other Ways to Get Started with Spawn

If you’d like to play around with Spawn locally, follow the Spawn Getting Started tutorial.

If you’d prefer to use Spawn with GitHub Actions, there’s a tutorial for that as well.

Thanks for the Great TSQLTuesday Topic

I’ve wanted to write this article for a while, but I kept not getting around to it. Thanks to Anthony Nocentino for the writing prompt.

The HBO Max Integration Test Email Means You Should Make Automation Safer: For Everyone

help@KendraLittle.com (Kendra Little) — Fri, 18 Jun 2021 16:15:47 +0100

When I woke up today in the UK, Twitter was alive with jokes, hot takes, and sympathy about an email sent out to millions of folks on a contact list for HBO Max featuring the subject line, “Integration Test Email #1”.

Here’s what the email looked like:

This email went out to more than subscribers: just about everyone with an email address got one. (Except me! I felt left out. My partner got one, so it’s in the family, though.)

What Happened?

The folks at HBO Max explained a little bit about how so many people were involved in this integration test:

No Really, What Actually Happened?

I give the folks at HBO Max credit for the supportive tone in their tweet. As someone who has had some big “oops!” moments with databases and production systems, I know the feeling of REALLY wanting to turn back time after you realize that you’ve accidentally caused things to go horribly wrong.

But it wasn’t “the intern”.

This only happens to an intern (or even to a Senior Engineer) if there aren’t safety measures in place. We all need safety measures: interns, engineers of all levels, and customers.

Two Key Pathways to Protect

Pathway 1: Restoring production datasets into development environments

How many of us have ever been at a company where we restored the production database to the development environment, and then at one point someone accidentally emailed all the customers?

I have. When I ask others this question at database-related events, lots and lots of folks raise their hands. This is an often-told story.

This happens all the time.

There are reasons why production-like datasets are necessary in dev and test environments. There are also important reasons to redact and/or mask sensitive parts of that data.

Redacting and masking sensitive data protects our staff from sending out “Integration Test Email #1” from the development environment. It also protects our customers. Development datasets are also a prime target for hackers who use phishing techniques to obtain credentials that allow them to harvest valuable data from development environments.

Pathway 2: Safety checks in pipelines

While it’s most likely that this email was caused by a production dataset being used in a development environment, it’s also possible that this was caused by an automated deployment pipeline with few or no safety checks.

Automation can either make things very unsafe, or very safe: it’s all in how your team implements automation.

Making automation pipelines safer doesn’t necessarily mean making them slower. With most automation systems, one can implement a combination of automated tests to catch problems. Also consider checks to estimate change risk, such as the number of emails which would be sent. You may choose to allow low-risk changes to flow more freely through the system, while higher-risk changes require manual review.

What to Learn from Integration Test Email #1

The main thing to remember is that this can happen to anyone. We all can be the one to make a mistake.

The key thing is to work towards making things safer, and increasingly making it less likely to make mistakes.

Take the time to:

Look at whether production data is being restored into development environments without masking of personal and sensitive data. If it is, start up a conversation about fixing that.
Identify the safety checks in your automated pipelines, and if there are improvements you can initiate here to make deployments safer for everyone.

Managing Drift in Automated Deployments to SQL Server Databases

help@KendraLittle.com (Kendra Little) — Fri, 04 Jun 2021 12:17:48 +0100

One big gotcha that teams often encounter when automating deployments for databases is that it’s difficult (or sometimes impossible) to ensure that all changes to the production database are performed through the automation pipeline.

These out-of-band changes case the production database to “drift” away from the schema as defined in version control.

Why does this happen, and what do you do when your production database gets modified outside of your automated deployment pipeline?

How Production Drift Happens

I find that there are three common causes of production database drift.

Process lag: Production drift is especially common when process changes are occurring and the automated deployment process is newly established. In this case, not everyone is quite on board with the new deployment procedure and changes continue to flow into production via older processes.

“Oops!”: Another common cause of production drift is accidental deployments. It’s pretty darn easy to deploy something to the wrong database. Sometimes the person who does this realizes their mistake, and sometimes they don’t.

“Whatever it takes” requirements: Some business critical databases rely on immediate remediation for certain types of issues to be performed by subject matter experts as soon as possible, and core stakeholders do not even wish to wait for a pipeline to execute to deploy corrective changes once they have been identified. An example of this could be an index being created to remediate a critical performance problem that unexpectedly appeared. (Note: many teams find ways to optimize their pipelines so emergency changes may be deployed incredibly quickly, but this varies.)

Automatically Reverting Drift Is Dangerous

Some folks like the idea of wiping out any drift when the next automated deployment occurs.

This can be done using “state” based deployments for SQL Server: one can do this when deploying Redgate’s SQL Source Control projects using SQL Compare (or an automated pipeline), or when deploying Microsoft’s Database Projects (SSDT). With this approach, code is automatically generated to apply the schema in the project to the target database. If anything has changed in the target database since the last deployment and it wasn’t incorporated into the schema in the project, it automatically be eliminated.

But automatically reverting drift is risky.

You can cause a performance regression - If an emergency fix was put into place for a critical performance problem, that problem is very likely to come back if we automatically revert the fix.

You probably didn’t test this change: No automatically generated code is guaranteed to be 100% perfect 100% of the time. If changes were made to production and not to other environments and you automatically generate and run code to revert it, that code probably wasn’t tested anywhere else. Do you really want that code to run?

You can lose data - In some cases, the drift might have done something like create a new column or table and persist some data there. Is it really safe to drop that column or table? We won’t know unless we take time to investigate.

Generally, we want to either prevent drift, or we want to be alerted to the drift and to investigate it.

Option 1: Alert When Drift Occurs

One of the best things that you can do is to monitor for database drift and alert when it happens. Establish a procedure to follow up on the alert:

Revert the change
Incorporate the change into your project retroactively

Grant Fritchey wrote up some sample code for monitoring for and alerting on drift in SQL Monitor. This approach uses an Extended Events Trace and SQL Monitor’s Custom Alert functionality.

If you’re using a different monitoring solution, it’s absolutely worth investigating if you can implement monitoring for database drift.

Option 2: Automatically Incorporate Drift into Your Development Environment

Another option is to regularly check for production drift and proactively pull it into your development environment. Ideally, this should be an automated solution or you are likely to miss the drift.

If you have a schema snapshot tool, this can be done by automating schema snapshots of the production environment and copying the snapshots to your development environments. Then run a comparison against your development database or project to check for drift.

For Redgate SQL Compare users, there’s even a tool called “SQL Snapper” that is designed to help make taking these snapshots more easily.

For others, Microsoft provides the ability to extract the schema from a database with SqlPackage.exe.

Option 3: Prevent Drift Altogether

One very popular use of DDL Triggers in SQL Server is to prevent accidental or out-of-process database changes.

These are a bit like a child-safety lock: you need to be able to disable the trigger in order to make schema changes. This means that it’s always possible for an adminsitrator to disable the triggers and make a change.

That’s generally fine, the cause of drift is often ignorance, forgetfulness, or accident rather than malicious users. But it does mean that this approach pairs well with also adding an alert to monitoring to check for drift, too.

Why NoSQL Database Adoption Is Not a One Way Journey

help@KendraLittle.com (Kendra Little) — Thu, 27 May 2021 08:34:12 +0100

A coworker shared with me recently that a customer is wholly investing in adopting non-relational datastores.

“Is NoSQL taking over?” they asked.

In my experience, organizations who go “all in” on non-relational databases end up with a mixed environment. In other words, they continue heavily using relational databases. They realize that they need to invest in doing well with every kind of database they use for mission-critical data.

At the Beginning of the Journey

Key team members and leaders get on board with investing heavily in working with non-relational datastores. Typically there are a few headline use cases where non-relational databases fit really well.

This gives a feeling that these types of databases will work well for every system.

Over Time, Some Projects Succeed and Some Fail

Generally, teams do find use cases where the non-relational databases work very well.

They also find plenty of cases where moving to a non-relational datastore adds complexity and reliability goes down.

Relational Databases Are Good at, Well, Relations Between Data

Relational databases provide well-documented, simple ways to efficiently store data and to join data back together to return meaningful results. It’s relatively quick and efficient to get a project off the ground using a relational datastore. Modern hardware and cloud computing patterns also make this more scalable than ever.

But this doesn’t fit every use case.

Sometimes it is better to have a database designed specifically to store and retrieve documents. Or to use specialized methods to store and query complex relationships using graphs. Or to use a simple key-value mechanism.

Because Different Types of Databases Are Suitable for Different Use Cases

Nearly every larger organization ends up with a mixed environment which includes plenty of relational databases. After the romance period is over, they realize they need to invest in processes, people, and tools for all of the technology in play.

My Sketchnotes from Redgate Level Up Week 2021

help@KendraLittle.com (Kendra Little) — Mon, 24 May 2021 17:39:17 +0100

Redgate regularly holds an internal conference called Level Up week. In 2020 and 2021, this conference has been scaled to include multiple Redgate offices using a remote learning approach.

While this is an internal conference, I can share my drawings and notes from sessions I attended. I hope these inspire you with ideas on how you can contribute to learning in your own organization, or perhaps simply inspire you to give sketchnoting a try.

Why Sketchnote?

I like sketchnoting for a couple of reasons:

Sketchnoting during a talk helps me focus more on the content and think more deeply as I listen. (I struggle to focus while sitting completely still.)
Looking at sketchnotes by other people often makes me interested in a topic which I might not otherwise look into at all. This often results in further reading on my talk, or deciding to attend talks or watch videos on those topics.

Basically, I find sketchnotes interesting and inspiring.

The Shape Up Product Development Methodology

Tom Walsh shared his learnings and experiences using the Shape Up working methodology.

This Product Development methodology was created and shared by by Basecamp in 2019. (From what I can tell, this isn’t at all related to current events at Basecamp, but dig in and have your own critique.)

Design Sprints

Redgate’s Tushita Gupta gave a terrific session with an overview of running a successful Design Sprint.

How to Delegate and Be Delegated To

We all know that delegation is a good idea. But most of us also don’t think enough about how to delegate well.

This workshop, co-taught by Jose Lima, got me really thinking about how to do better with delegation, whether I’m the delegator or the delegatee.

Programming with Types

Jeremiah Peschka gave a talk about using types build intelligence deep within a program.

Remote Workshops That Don’t Suck

Neil Turner gave an excellent meta-workshop on giving workshops.

I don’t know about you, but I personally prefer remote workshops to in-person workshops. However, all workshops are hard work: they can suck whether in-person or remote. I thought these were great tips on the remote variant.

I also really appreciated that the first question Neil had us start with was asking us to always consider whether a workshop is the best option, or if perhaps this is a meeting that might be an email. (Or a document co-created asynchronously. Or a mural board contributed to asynchronously. Etc.)

Thanks Redgate for a Great Level Up Week

I feel very grateful to get to attend and participate in such a terrific conference each year. Thanks to all the Redgaters who sponsor, organize, and participate in this event.

Azure Static Web Apps Are Now Generally Available

help@KendraLittle.com (Kendra Little) — Tue, 18 May 2021 07:31:05 +0100

I wrote recently about moving my blog from Wordpress to an Azure Static Site.

Azure Static Web Apps have now moved out of Preview and are now generally available. A free tier has been announced which is great for personal blogs.

A Free Tier

Here’s a screenshot from https://azure.microsoft.com/en-gb/pricing/details/app-service/static/ today:

Example free tier limits and pricing – May 18, 2021

New Features

New features were announced at the time of GA.

For casual bloggers who are getting started, the biggest among these is probably that Azure Static Sites now has full support for root domains — no www or other subdomain is required anymore.

Happy blogging!

Maintaining Balance While Retraining in a New Professional Area

help@KendraLittle.com (Kendra Little) — Mon, 10 May 2021 00:00:00 +0000

It’s common to shift your professional focus multiple times over your career while working in tech. But moving into a new role often causes a lot of stress.

In my recent lightning talk for Mental Health and Awareness Day, I reflected on the lessons I’ve learned to set myself up for success and protect my health when taking on a new professional challenge.

I’ve Transitioned Between a Variety of Roles Over the Course of My Working Life

Looking back at my working hours, my roles have included working as a: Clown, Craft Store Clerk, Office Temp, Data Specialist, Tech Support, Build Ops Engineer, Database Administrator, Consultant and Small Business Owner, Developer Advocate

Recently I’ve transitioned into a new role: Product Manager.

I Enjoy New Challenges, but the Early Stages Are Quite Tricky

I am a person who likes to contribute. This means that when I move into a new role with a steep learning curve, it’s generally unclear how I can best contribute and it can take a while to get up to speed.

For me, the steeper the learning curve, the greater my stress and uncertainty.

However, I’ve found that career shifts help me grow quite a bit, and are well worth doing!

Over the years, I’ve learned to recognize where I tend to go wrong and to recognize how I can avoid these errors. This is a continual process, but one which I believe is worth the investment.

Recognizing Problem Patterns and Building Good Habits to Compensate

Problem pattern: Focusing on the negative. Change makes it trickier to recognize my own wins. Bad days will happen in any role, but it’s easy to blame myself for them when I’m in a new role.

Getting perspective: I’ve found that having a sounding board at work is invaluable. If I can find a peer or colleague who is available for regular checkins, this can help me gain perspective on how I’m doing. It can also help me avoid catastrophizing bumps in the road.

Problem pattern: Overworking. It also becomes very easy to work very long hours to try to get up to speed quickly. This tends to be one of the greatest traps I fall into. Working long hours feels helpful initially, but over time it tends to decrease my creativity and wear away at my prioritization skills.

Setting boundaries on time: I’ve found that working predictable, defined hours makes me better at my job. Achieving this involves not only stopping work at the right time consistently, but also resisting the urge to look at emails and work channels during off-hours.

Problem pattern: An urge to be irreplaceable. As someone who loves to contribute, it’s easy for me to fall into the trap of finding an area where I can offer unique value– and not taking the time to empower my coworkers to offer this value without me. In my experience, if something is worth doing more than once, it’s generally worth either automating, or empowering others to do it!

Working as a team: Prioritization and identifying the more important areas to contribute to are key to most roles that I’ve had. Practicing prioritization early in a role and validating these choices with my managers and my teammates is the best thing I can do to get started. Good prioritization also helps me make sure that I don’t get stuck in an area of repeating tasks, and can either automate or share the ability to take care of things with my colleagues.

A List I Revisit Often

I don’t think I’ll ever find taking on new challenges easy. This will always be something that I work at improving. I find it well worth the work, however, and the work is lighter now that I’ve recognized where I tend to go wrong, and the habits that help me do well:

Set boundaries on working hours
Disable work notifications or remove phone apps
Say “no” to work activities you can no longer do
Pause on extracurriculars if you need that time to relax
Ask a colleague to have regular check-ins
Practice recognizing feelings of self-doubt
Practice acknowledging your progress
Ask for help and delegate

Writing This Lightning Talk Was Helpful

One of the great things about public speaking is that it drives both learning and personal growth. These days, I’m careful about taking on too many speaking engagements and conferences in light of what I’ve said above about prioritization. However, this event and this topic came and just the right time as I’m finding my way as a new Product Manager.

How I Got Twitter Cards Working in Hugo with the Mainroad Theme

help@KendraLittle.com (Kendra Little) — Fri, 07 May 2021 00:00:00 +0000

Hugo now has native support for Twitter Cards, but it took a few steps to get them working on my site in the way I wanted.

Here’s how I configured Twitter Cards in Hugo without any new shortcodes.

What’s a Twitter Card?

When you share a link on Twitter, it sometimes automatically adds an image to the tweet with a title and link at the bottom. This is a Twitter Card.

A Twitter card featuring a drawing of our beloved dog, Mister Little

Back when I used WordPress, I had a habit of setting the “featured image” for a post. Among other things, this featured image would typically be shown on the Twitter Card. After I switched from WordPress to Hugo, I noticed that when I tweeted links to my site, those pretty little cards weren’t showing up with my current configuration.

I like to be active on Twitter, so I wanted to fix this.

Twitter Card Validator and Metadata in HTML

If you want to check out a Twitter card (or the lack thereof) without actually tweeting, Twitter provides an online Twitter Card validator.

Also, you can inspect a page’s HTML and look for metadata about Twitter cards. This can be especially useful if you’re working with your local webserver (the Twitter card validator can’t do anything with “http://localhost:1313/” because it has no way to see your local webserver).

You can “view page source” from your local webserver and look in the head portion of your HTML for meta tags that start with “twitter:”. Here is the way this metadata shows for this post on my local webserver, as I’m authoring it.

Mistakes I Made in config.toml

The Mainroad theme supports Twitter cards. When starting out, I had identified the twitter_cards setting in my config.toml file and I had set that to true. But my cards weren’t working. Why?

One reason was that I was not specifying a default image to use for a Twitter card. Here is the default images setting that worked for me:

One mistake I made along the way: I accidentally omitted the square brackets. That didn’t work at all, and it took some time to notice my mistake.

You Can Manage Twitter Cards in Detail via Front Matter in Hugo

I like having a default TwitterCard, but I also like managing it for each page. Here is the Front Matter of this post as I’m authoring it in VSCode (or as I think of it, the header).

This might look like a lot of things to remember to list at the top of each post, but it’s not hard at all – I have an “archetype” set up in Hugo which makes auto-populating this list easy when I create a new post. (More on that below.)

I like managing these details on each page because it makes it simple for me – I can see what the thumbnail is at a glance, if I have it set to the same thing as the Twitter Card or something else, etc. It’s really easy to change at any time.

Updating the Front Matter of 500+ Pages Using a Regular Expression Find and Replace

I have more than 500 pages on this site. Wherever I have a thumbnail listed, I wanted to update the front matter on each one to list the Twitter card and default it to the same image as the thumbnail. (Maybe the thumbnail is supposed to be used as the Twitter card image by default in Hugo if the card image isn’t specified, I’m not clear– but it wasn’t working on my site and this seemed like the simplest fix if I could get it done fast.)

I was able to do this easily using the “Find and Replace in files” feature in VS code, with regular expressions enabled on my search and by making use of an “index” in the search.

I searched in files for: ^thumbnail: (.*)\n

I replaced this with: thumbnail: $1 \nimages: [$1]\n

I found this post very helpful in figuring out the regular expression to use here.

Updating My Archetype File

As I mentioned above, I use an archetype in Hugo. This is essentially a template which will help me quickly create a new markdown file for a post with all the front matter I want pre-configured for me to fill in.

I updated my archetype file for posts. Here’s what I currently have in the file, which in my case lives at archetypes/posts.md:

---
title: "{{ replace .Name "-" " " | title }}"
date: {{ .Date }}
categories:
-
tags:
-
description: "add description"
toc: false
thumbnail:
images: []
draft: true
GHissueID:
---
Add post here.

To create a new post with the archetype, I run the following from the command line in my Hugo project, with whatever I want as the post title as the markdown file name:

hugo new posts\hi-i-am-a-new-post.md

That gives me a nice file to fill in with my post.

This was a fun one to fix

There might be a more elegant way to solve this rather than that regular expression I used– but I really enjoyed it!

How to Enable TCP/IP in SQL Server Developer Edition, Even If the Configuration Manager Is Missing

help@KendraLittle.com (Kendra Little) — Wed, 05 May 2021 17:46:08 +0100

SQL Server’s Developer Edition installs with a different default network protocol configuration than Standard Edition and Enterprise Edition.

Fixing this has gotten more complicated because it’s become common for the SQL Server Configuration Manager to not be installed on a developer’s workstation in an easily accessible way.

Many Editions of SQL Server Install With TCP/IP Enabled, but Not Developer Edition

If you don’t always work with Developer Edition, it’s easy to forget that it has a different default for the TCP/IP protocol.

This isn’t a bug – Microsoft has documented this (highlighting mine).

Edition	New installation vs. previous installation is present	Shared memory	TCP/IP	Named pipes
Enterprise	New installation	Enabled	Enabled	Disabled for network connections.
Standard	New installation	Enabled	Enabled	Disabled for network connections.
Web	New installation	Enabled	Enabled	Disabled for network connections.
Developer	New installation	Enabled	`DISABLED`	Disabled for network connections.
Evaluation	New installation	Enabled	Enabled	Disabled for network connections.
SQL Server Express	New installation	Enabled	Disabled	Disabled for network connections.
All editions	Previous installation is present but is not being upgraded.	Same as new installation	Same as new installation	Same as new installation
All editions	Upgrade	Enabled	Settings from the previous installation are preserved.	Settings from the previous installation are preserved.

But increasingly, developers do want to connect to Developer Edition SQL Server instances over TCP/IP. Maybe those development instances are being used for a build or for testing purposes. Or maybe they are simply using a development tool that requires TCP/IP.

To fix this…

1. First, run the program “SQL Server Configuration Manager” (or find it in Computer Management)

Option 1: The traditional way to fix this is in an application named “SQL Server Configuration Manager”, which is (sometimes) installed when SQL Server Developer Edition is installed. You can find this in the Windows Start menu or search bar. If you don’t see the SQL Server Configuration Manager installed…

Option 2: Run the “Computer Management” app. This can be started from the Windows Start menu or search bar. The command line to start this is compmgmt.msc.

Expand “Services and Applications” at the bottom of the left menu.

If you have more than one version of SQL Server installed, you will see more than one “SQL Server Configuration Manager” snap-in available. ou generally want to choose the highest version, but to tell which one that is, you will need to click into “SQL Server Services” in each one and see which one lists the most SQL Server instances.

Now you have all the functionality of the SQL Server Configuration Manager in front of you. (It’s just this snap-in.)

Thanks to Chrissy LeMaire for this tip about Computer Management in the comments. I had no idea!

Option 3: If Computer Management doesn’t work for you for some reason, you could also add snap-ins to the Microsoft Management Console:

From a terminal or the windows search bar, start the Microsoft Management Console by running: mmc.exe
Go to “File”, then “Add/Remove Snap-in”
Find “SQL Server Configuration Manager” in the list of available snap-ins and click “Add” to move it to the right
Click “OK”

If you have more than one version of SQL Server installed, you will see more than one “SQL Server Configuration Manager” snap-in available. You generally want to choose the highest version for the reasons above. This may take some trial and error. You have the option to “Save” your Microsoft Management Console configuration (under the File menu) so it’s easier to re-open your configuration the next time you run MMC.exe.

2. Next, enable the TCP/IP protocol for the instance

Expand “SQL Server Network Configuration
Click “protocols for <the name of your instance>”
Find the TCP/IP protocol in the details pane, right-click it, and select “Enable”

3. Finally, restart the SQL Server Instance

You will see a message that the SQL Server instance needs to be restarted for this to take effect.

Go to “SQL Server Services”
In the right pane, right-click the instance to restart
I typically choose to stop it, wait a beat, and then start it separately – just so if anything fails I know what bit it was on when it hit the problem.

But, Why?

This configuration default has never actually made much sense to me, personally. I’ve always worked in environments where it’s been useful to allow remote connections to developer SQL Server instances.

I’ve heard the argument before that it’s more secure to default an instance to this configuration and only enable TCP/IP connections if shared memory / Named Pipes aren’t sufficient. But if that was the case, wouldn’t this be the default for all editions? :thinking:

In any case, this default has been in place for a long time, and I think it’s unlikely to change anytime soon.

Moving From WordPress to an Azure Static Site with Hugo

help@KendraLittle.com (Kendra Little) — Mon, 03 May 2021 00:00:00 +0000

I’ve moved from WordPress to an Azure Static website built with Hugo. The conversion, including grooming all my old posts and a special setup for my free courses took up most of my personal time for a week (including a 4 day weekend).

Here’s why I made the change. I’ll also share the basic components I chose for my site, the tools and steps I used in the conversion, and some lessons learned from the implementation.

This is going to be a long one, so buckle up! :seat:

Why I Ditched WordPress After ~12 Years

I used WordPress for blogging from 2009 to 2021. Even after some cleanup, I have more than 500 blog posts. So I had a good amount of investment with WordPress. I knew that changing how I manage and publish my blog content would be a lot of work.

But I had significant pains with WordPress:

High Cost: I used an expensive WordPress host when I was running my own business but no longer wanted to pay the big bucks for hosting when I made my video content free and my site returned to “hobby” status. There are additional costs with WordPress for plugins for things like backups, security scanning, and flexible themes.
Bad Performance: When I downgraded to a cheaper host, my performance tanked. My site was often so slow that it was hard to tell if it was online or not, even with Cloudflare supposedly set up.
Low Portability: It took many hours to even get my WordPress site migrated to the cheaper host– just figuring out how to get the backups I had taken to work with a migration took hours with support from the backup tool company. (Turns out most of the tooling is optimized to restore to the same host, but that doesn’t mean you can’t keep trying to do it and having it fail :facepalm:.)
Significant Complexity: Once it was migrated, troubleshooting the performance issues with the new host was the cause of a lot of frustration. At times there was clearly an issue with underpowered hardware, but nobody could help me figure out why everything just generally sucked. Maybe there was some misconfiguration in my setup, but with a MySQL database, webserver, plugins, themes, Cloudflare, and who knows what else was going on at the host involved – I just had no interest in troubleshooting it.
Security Hassles: The more complex a system is, the harder it is to prevent vulnerabilities. It’s fairly common for WordPress sites to be hacked and have malicious code inserted to advertise for spammy links – this happened to me once a few years back.

I Considered Throwing My Whole Site Away

I wasn’t having fun with the blog anymore after I’d migrated to the Terrible WordPress Host. Troubleshooting WordPress wasn’t appealing. I didn’t want to spend a bunch of money to have someone else do it, either. I considered abandoning the blog and simply moving on. :wastebasket: :raccoon:

The main reason I didn’t do that was that those friends in the community who DM’d me about my site being down mentioned that they found my content useful. This feedback helped me make the right decision. Thanks, friends. :revolving_hearts:

So I looked around to see if there was a newer way to blog which would be more fun for me, and which wouldn’t have all these problems.

Why a Static Website?

As someone who spends most of her time thinking about databases, I didn’t even know what a static site was until I started looking into new options for my blog.

In the early days of the internet, all web pages were pretty simple: just some HTML. Eventually with some CSS styling. Static websites are a return to this concept: when your browser loads a page on a static site, it’s mostly just loading pre-baked HTML and CSS.

This contrasts to dynamic websites, where pages may load different things for different people. There are also generally more types of interactions available on dynamic sites. But with the flexibility of dynamic sites comes the complexity and potential for related issues I mentioned above.

There are now a few open-source static website generators which are very popular both for blogs and other types of websites. The most popular are Jekyll, Hugo, and Gatsby. The main benefits of using a static site built with one of these generators are:

Low or No Cost: There are quite a few low-cost or free hosting options for static sites, including GitHub Pages (free), Azure Static Web Apps (Update May 18 2021: now Generally Available with a free tier), and free plans on providers like Netlify. Static site generators like Jekyll and Hugo are typically open source and completely free to use. An active community of users of these static site generators have created a large amount of free themes, many of which are eminently customizable.
High Performance: By pre-building the content and minimizing what the client browser does when loading pages, static sites can offer very high performance and fast page load time, even while having lower costs.
“Shifted Left” Complexity: Using a static site generator can seem quite complex if you aren’t used to working with a bit of code, version control, and cloud tooling. But if these things are appealing to you, the complexity is all in a learning curve. Once you’re on board, the normal process of blogging becomes quite simple, and the static site itself is generally quite simple in how it operates.
High Portability: Having the contents of your site all in source code gives you so much more portability than you have by using a content management system which keeps much of the content in a database. I am a huge fan of databases – I’ve spent my entire professional career thinking about how to best work with databases! But I don’t have a “golden hammer” view of databases. It makes much more sense to me that my personal blog just involve a bunch of text files and images which are checked into version control. This is easier to manage and much more portable long term. Not only can I move my static site easily between different hosts by simply deploying elsewhere, I also have a lot more options if I want to switch to something different later on.
Better Security: Static sites are generally much more simple than dynamic sites and have a lower surface area for attack. Also, as long as the repo holding my code is secure, I always have the option of clearing out my live environment and redeploying everything from scratch if I need to.

The downsides of static sites are that they are less accessible to people who prefer to work in a lower-tech fashion.

It also can take a little extra work to include interaction in your website and often involves integrating third party services into your site. You’ll see that I have workarounds for blog comments, newsletter subscription, and a comment page below, all of which leverage an external provider.

Why I Chose Hugo and Azure Static Web Apps

I didn’t have a tiny project – remember, I had ~500 blog posts to migrate and a desire to do a custom setup to still offer ~30 free video courses online. Also, the performance of my current WordPress site was just abysmal. I frequently got DMs on Twitter from friends nicely letting me know that my site wasn’t working :cold_sweat:.

It felt advisable to ask my network for advice.

My first thought was that Chrissy LeMaire had written about migrating her WordPress site to GitHub Pages. But when I mentioned this on Twitter, Chrissy advised that she might approach this a different way if she had it to do again:

Good intermediate solution but not a good final solution! I haven’t blogged much since then because it’s a pain 😭 In retrospect, I’d look for a WordPress to Markdown plugin, and put the work in upfront.

My next thought was… hey, didn’t my partner do something like this for his blog a while back? Uh, yeah, I thought of Chrissy first and my partner second. :woman-shrugging: Is that weird?

Jeremiah explained that he had fully converted his site up front (like Chrissy recommends). He is using Hugo with the Mainroad theme and deploying to an Azure Static Web App. He recommended this approach and said that his experience now is good. But he also mentioned that the tooling he used to convert his posts wasn’t terrific and that he’d look for another alternative for getting WordPress content into Markdown.

Coincidentally, Justin Bird mentioned on Twitter that he also moved to Hugo for his site. He said that he had used lonekorean’s WordPress-export-to-markdown project to convert his content.

Together, this was all enough to give me a good starting point, so I dove right in.

In other words, I didn’t do an exhaustive comparison of static web generators and hosts. From what I’ve read, there are multiple great choices here. I’m already interested to try Jekyll and Netlify in future projects, perhaps.

Overview of My Static Site Setup

I’ve alluded to a lot of this so far, but here’s a summary of all the tooling and services I am using for this site, along with cost considerations:

LittleKendra.com is now an Azure Static Web App. This is currently in preview and it’s totally free, but there may be a cost for this in the future. (Update May 18 2021: Azure Static Sites are now Generally Available with a free tier)
My DNS is hosted by Cloudflare, which is also proxying traffic. This is free.
I use Hugo as my static site generator with the Mainroad theme. This is all free.
My code is in a private GitHub repo. That’s free.
I use VSCode to author changes for my Hugo project, commit to Git, and push up to GitHub. VSCode is free.
When I push new commits to my GitHub repo, a GitHub action runs to automatically build my code. If successful it deploys to my Azure Static Site. A good amount of GitHub Action usage is free .

And here are the third party integrations I’m using:

I’m using TinyLetter to support people who want emails for blog posts (see below for details). This is free.
I’ve embedded a Google Form in a simple contact page for people who want to send me a message (see below for details). This is free.
I’m using the GitHub issues API and a public repo for blog post comments (see below for details).This is free.
I have a large amount of YouTube videos which are embedded throughout the site via Hugo shortcodes. Also free!
I also have a few GitHub Gists which are embedded throughout the site using Hugo shortcodes. (For shorter code samples, these are in the posts themselves and formatted via Markdown, no extra functionality needed.) Totally free.
I’ve set up some ifttt applets to see new entries in the RSS feed. One automatically tweets while the other sends the info to my free Buffer account, which will automatically post the info to LinkedIn. Free.
Giphy hosts some animated gifs which I created as short demonstrations for some posts. These are embedded simply as images. (More info on this below.) Free as well.

The Conversion Process

Set Up My Azure Static Web App, My Hugo Project, and Hook Up Publishing

I began by following the steps outlined in this excellent Tutorial: Publish a Hugo site to an Azure Static Web Apps Preview. This walked me through:

Creating my hugo project
Adding a theme (this is installed as a git submodule)
Setting up my Azure Static Web App in the Azure Portal
Deploying my Hugo app to my Azure Static Web

I carried all these steps in VSCode, the Azure Portal, and the GitHub web interface. This was all really accessible – the steps even automatically created my GitHub Action which does the publishing! :star:

Your Azure Static Web App is created with a default name. This enabled me to keep working on my project for a while and convert my existing content, then finally redirect my domain name when I felt ready to do so.

If you don’t have existing blog content to migrate, at this point you can just register a domain name, point it at the static site, and you’re off to the races!

Convert My WordPress Content Into Markdown and Copy the Files Into My Hugo Project

I used lonekorean’s WordPress-export-to-markdown project to do this conversion, as Justin Bird recommended. I had a good experience with this script.

The instructions for the script guided me through this basic process:

Installed Node.js
Exported “All content” from WordPress – this generates an XML file you save locally
Cloned down the WordPress-export-to-markdown script from GitHub
Initiated the wizard from the command line and stepped through it

During the conversion process, the script will go to your live WordPress site and save off images referenced in your posts. You can throttle the image downloads if you hit errors. I found that it did a great job in general of grabbing my images. It consistently failed on images which I’d imported from a different WordPress site at some point, for whatever reason, though. It was a small enough amount that I cleaned these up manually when I “groomed” my posts.

I ran through the export process a few times and iteratively copied the files into my Hugo project and tested to make sure the generated files matched my old WordPress URL structure. It’s great that the script is flexible about this!

In my case, all of my old blog URLs are in the format: kendralittle.com/yyyy/mm/dd/the-post-url/

I didn’t use any year or month folders – I put all my posts in a single folder and all my images in a single folder for simplicity
The conversion script automatically put the post’s date, categories, and tags into the header of each markdown file
I used the Hugo setting below to structure the URL format for my blog posts – this picks up the date from the markdown file’s header and uses it in the URL structure for the published post

[Permalinks]
posts = "/:year/:month/:day/:filename"

My basic folder structure in Hugo (excluding my courses, more on that later) looks like this:

.
├── content
| ├── posts
| | ├── a-post-title.md // <- https://www.kendralittle.com/1995/01/29/a-post-title/
| | └── i-wrote-this-too.md // <- https://www.kendralittle.com/2010/10/01/i-wrote-this-too/
| | └── {etc} // <- ~500 files total (one per post)
| └── about-kendra-little.md // <- https://www.kendralittle.com/about-kendra-little
| └── {etc} // <- I just have a few pages, no biggie
├── static
| └── images // <- All images
└── themes\Mainroad // <- This is the theme, which is a git submodule

Test How Everything Looks Locally

One thing which I love love love about my static site is that I can do loads of work locally. All I need to do is run this command line in my project:

hugo server

This starts up a local webserver at http://localhost:1313/, which I can load in a browser and see exactly how my site looks now.

And even better, every time I save a change to a file in my Hugo project, it rebuilds and the change is immediately visible on the webserver. And it’s :boom: FAST :boom:. Having this super-fast local build and deploy made working on the conversion process really fun for me. I got to see things transform right before my eyes.

Groom My Converted Content, Keep Testing Locally, and Periodically Push to GitHub

No conversion script will be perfect for every use case. I used a combination of manual editing and “find and replace in files” functionality in VSCode (at times with regular expressions), to get my content into a shape which I like.

Things which I fixed up include:

Replacing YouTube links with Hugo shortcodes. I had at least three different styles of YouTube links and embedding styles in my old posts so this took a little work.
Fixing up code samples in blog posts. I have a lot of posts with SQL scripts which had been formatted in WordPress by at least three different plugins over the years. These are now very simply styled in markdown. This was probably the most time consuming thing for me to fix given the amount of code I had, but it was very satisfying.
I had a few tables in posts which needed to be converted into markdown. I still had my old site available so this was just a matter of finding the posts by searching, loading up the old page, and using an online converter from HTML to markdown to generate the code.
Identifying those images I had which didn’t get converted and replacing them as needed. Often these had a strange issue with links to a site which no longer exists being wrapped around them which needed cleaning – almost certainly a remnant from having exported/imported them between WordPress sites before.
Cleaning up places where I had linked an image to the source file in WordPress (these links all fail now because the folder structure is different, and I’m not sure why I even used those links in the first place)
I had some large animated GIFs in posts. I moved these into Giphy and embedded them in Hugo as an image using the source URL from Giphy

![actual-elapsed-time-demo](https://media.giphy.com/media/VYWgrTTqR91933pz3f/source.gif)

Throughout the process of fixing things up, I constantly looked at my local webserver as I saved things.

I also committed fairly frequently, and periodically pushed a batch of commits up to my GitHub repo to make sure that the build and deployment from that point was still working.

Move My Free Training Courses Content Over

After working with Hugo for a little while, I became ready to learn how to create a page structure alongside the blog to hold my courses and lessons.

I had used a plugin to manage my courses in WordPress, and happily the WordPress-export-to-markdown script did convert markdown pages for all my courses and lessons. I just needed to figure out how to get these to display in some coherent fashion without manually having to add a lot of links to list out lessons for each post and navigate between them– and ideally not change too many URLs.

It turns out that Hugo is very much suited to helping build out content like a “courses” structure with lessons. The main things I needed to learn about were:

Page bundles in Hugo - Branch bundles allow me to have a main course page in which all the lesson pages are auto-magically linked into the parent course page in order.
Taxonomies in Hugo enabled me to create a custom taxonomy for course categories, and drive navigation that way. There’s lots of documentation, but I found this post by Jake Wiesler really helpful in getting me going.
Templates to customize the look and feel of different pages

I did a lot of searching and employed a great deal of trial and error, but having a really fast and convenient local build and webserver experience made this fun to work out.

Re-Point DNS and Go Live

My old site’s performance was so bad that I probably should have cut over nearly right away and just put a note on the new site that I was working on cleaning things up. But I was excited enough about what I was building that I wanted it to be a certain level of “done” before I showed it to people.

The process of cutting over is relatively simple, but I learned some important things about how to do this with an Azure Static Web App.

Microsoft has provided good documentation on how to setup a custom domain in Azure Static Web Apps Preview. Follow this, but note:

Azure Static Web Apps only support mapping a subdomain while in Preview. In other words, I could use https://www.littlekendra.com. The Preview does not support using a root domain like https://littlekendra.com (but it sounds like this is coming later). <– Update May 18 2021: root domains are now available
If you’re not a DNS wizard (and I am not), this can be confusing to configure. For simplicity, I pointed my nameservers to Cloudflare and used their tooling to configure my DNS. In my experience they do a good job simplifying DNS management on their free plan.

If you are using Cloudflare, follow the steps in the documentation above to configure CNAMEs, but also know…

There is a step in the Azure portal setup where it needs to verify the CNAME. For this to work, you need to have the CNAMEs set up in cloudflare set to “DNS only”. After the verification is complete you may change them to proxying traffic.
I needed to go to “Rules” and create a Page Rule to handle existing links to pages at the root domain and send them to the www version. Here is the rule for my site:
- If the url matches: kendralittle.com/*
- Then the settings are:
  - Forwarding URL
  - 301 - Permanent Redirect
  - https://www.kendralittle.com/$1

DNS takes time to propagate, and in this case waiting was the hardest part. I was so excited to go live with my revamped little site!

Tips and Tricks for Setting Up a Blog With Hugo

Here are some highlights of things I’ve learned and resources I’ve used while configuring my new Hugo site with the Mainroad theme. I found that there’s a really great community out there and that I could solve most of my problems with searching and testing out different options.

Mainroad Theme

There are only two things I’ve found which I couldn’t easily override or modify outside of the Mainroad theme submodule. I’ve worked around both of these by forking the Mainroad repo in GitHub and making the forked repo my sub-project. This allowed me to:

Control thumbnail display. I hate it when the post’s thumbnail image is automatically displayed at the top of a blog post – I’ll add it in if I want! I copied the code for this from an existing PR on the Mainroad theme
Replace the default site favicon. This appears to simply not have been made easily configurable but it’s easy to replace the file in my forked copy of the Mainroad repo.

General Styling

I use a lot of right aligned floating images. I couldn’t find a native way to do this, but used this undocumented cheat in my markdown code with a bit of css to make it work.
- The Mainroad theme has an option to configure a location to place your own css file to override the theme’s default rules, so this was pretty simple.
Not all themes support nested menus. Mainroad is one of the themes that doesn’t. I used this post Add Sub Menu in Hugo Website by Coding N Concepts to implement a sub-menu.
Social share icons on blog posts. I used another article from Coding N Concepts for this.

Contact Me Form

I spent a lot of time finding hard ways to do this before finding an incredibly easy way to do it. The easy way:

Create a Google form with required fields for email address, name, and message (free with any google account)
Create a custom shortcode that wraps the form in an iFrame in a Hugo page.
Call the shortcode from a Hugo page

It’s absolutely possible that trolls or bots could strike and hit the form with a lot of garbage data – but this is honestly a risk for any contact form solution. If that happens it’s really easy for me to just delete the form. For a 1.0 this is just a really easy way to start.

Blog Post Comments

A lot of people seem to use Disqus for their comments. You can sync your older comments from WordPress into Disqus if you want an automated way to carry those over.

As a woman on the internet, I have had a mixed experience with comments. Some very positive, some… not.

I decided that for now I am going to use a loosely integrated system with comments implemented via GitHub issues, as described here. I migrated over comments for some of my most popular posts according to Google Analytics, but generally didn’t worry about the rest. This means that comments are effectively closed on a lot of my older posts and, well, I’m totally fine with that.

In the future, I may look at implementing GiTalk. Justin Bird has implemented them here and they take using GitHub issues for comments to another level.

RSS and Blog Post Subscriptions

Some folks like to subscribe to blog posts – the two popular choices seem to be by RSS feed and by email.

It’s worth setting up both, if you have time. The email solution will depend on your RSS feed, so do that first.

Hugo provides native RSS feed capabilities, but it wasn’t enabled by default in my theme. I just needed to figure out what to put in my project’s config.toml file to make it work.
- This post on custom RSS feeds in Hugo from Coding N Concepts helped.
I’ve set up a Tiny Letter account to allow people to subscribe to blog posts. This is free.
- I use Zapier to automatically communicate to Tiny Letter when a new blog post appears in my RSS feed.This backendology post steps through how to configure this. Zapier has a free plan which should meet most people’s needs for this use case.

Closing Thoughts

I’ve had a great deal of fun working on this project. I’m more excited about blogging than I’ve been in a long time, and I’m looking forward to finding little ways to tweak and improve this site over time as well.

In a way, I guess this is all thanks to that terrible WordPress host. They ended up really improving my website, even if it was by convincing me to abandon WordPress and use static site generators for most of my personal website uses in the future. :woman-shrugging:

More genuine thanks go to the folks who helped me figure out this whole static site thing – everyone whose articles I’ve linked to above, plus:

My partner Jeremiah - for helping me curse at Git until it did what I wanted and letting me copy his DNS setup and forwarding rules
Justin Bird - for all the tips and tricks
Chrissy LeMaire for sharing her lessons learned

Help Me Not Be Selected to Speak at the Dativerse Virtual Conference

help@KendraLittle.com (Kendra Little) — Thu, 22 Apr 2021 00:00:00 +0000

A call for speakers is open for the very first edition of the Dativerse conference, which will be held on August 13, 2021: http://dativerse.io/

An Overview of the Event From the Dativerse Call for Speakers

Dativerse is a new event in the Microsoft Data Platform space. Our session selection criterias are easy…

Your session must be related to the Microsoft Data Platform

You contribute to the diversity of the Microsoft Data Platform We try to elevate those that are usually underrepresented at conferences due to their race, gender, sexual orientation, location or any other reason. We aim to create a safe environment for everyone and welcome seasoned speakers and newcomers alike. Should you require a mentor, we will make sure you will get all the help you’ll need to deliver your session!

I’ve submitted a session, but I’ve mostly done that because I want LOTS of people to submit sessions. I can now say, “I submitted a session and I think lots of others should, too.”

Why I’d Love for Many Others to Submit

I am a white, cisgender woman who is married to a man. I’m also at a point in my career where I’ve established a strong network. While we have a long way to go to reach gender equality in tech, I also recognize that I am relatively privileged due to my race, orientation, social background, and my connections. I feel it’s important for me at this time in my life to get better at helping others rise up.

I’ve also found in my life that the most exciting and productive teams I’ve worked on have had team members from a very diverse mix of backgrounds, experiences, and preferences. I strongly believe that having a more diverse, open, and welcoming community is good for everyone who is part of that community.

Help Spread the Word Far and Wide About This Event

Please encourage others from a rainbow of backgrounds to apply. My hope is that there will be so many sessions submitted from people of diverse backgrounds that my session is not chosen, and I can support the event in another capacity.

Please share and encourage others see if Dativerse is a good fit for them at this point in their career.

Free Virtual Event: Mental Health and Awareness Day on May 7th

help@KendraLittle.com (Kendra Little) — Wed, 21 Apr 2021 00:00:00 +0000

I’ll be giving a 10 minute lightning talk at the upcoming Mental Health and Wellness day event on May 7, 2021, hosted by the Data Platform WIT group.

My Session: “Maintaining Balance While Re-Training in a New Professional Area”

It begins at 1:10 PM GMT / 9:10 AM Eastern.

I am inspired to talk on this topic because I’ve shifted my career a few times:

From Tech Support to Database Administrator
Then to consultant and small business owner
Then to Developer Advocate
And now I’ve moved into a Product Manager role

These kinds of career shifts are common and healthy in the modern workplace. Moving into a new role often requires quite a bit of new learning at a fast pace, though, and for some of us this can be quite stressful. I’ll share the top lessons that I’ve learned to set myself up for success and protect my health when taking on a new professional challenge.

Can’t Join Live?

I believe my session will be recorded and made available on YouTube afterwards if the technical stars align.

There’s many more great sessions in the lineup– more info over on the Mental Health and Wellness day event page.

Ten Ideas to Improve Online Tech Conferences

help@KendraLittle.com (Kendra Little) — Wed, 18 Nov 2020 00:00:00 +0000

Amid the global pandemic, online tech conferences have had a natural surge in popularity. I’ve attended quite a few of these in 2020 across several technical areas (database tech, DevOps, privacy, tech research). The conferences have been both large and small, both free and paid, and have been held across a few different regions and time zones.

Many things have gone well. Organizations who are used to doing in-person events have worked hard to quickly adapt to online platforms, and I’ve seen some fantastic content. This is terrific!

But there are a few critical things that haven’t adapted well in the transition to online events. I’ve seen some major problems occur consistently across all these tech conferences. This isn’t due to laziness or lack of trying – it’s simply that some patterns that work at physical events don’t work well at online events.

We are learning as we go.

With creativity we can massively improve the experience of online events – and that’s really worth doing! Please help brainstorm and share your own ideas about how to make online conferences more awesome.

Why It Matters: If You Care About Inclusivity, Improving Online Conferences Is Very Important

My favorite thing about online conferences is that they are generally far more accessible and open than in-person events.

Travel and hotel costs for in-person events are often significant, and only those who are already in quite Senior roles typically have the benefit of these being paid by their employer. Ticket prices for in-person conferences are also generally higher, and less privileged community members similarly have a more difficult time getting these funded, often needing to pay out of their own pockets.

I also believe that low cost and free online events may create more open communities, as there is relatively low risk for someone new to the community to try out the event for a short time. Online spaces may feel safer for members of marginalized groups in terms of testing the waters to find out what a community is all about. This can result into bringing fresh ideas and many new people into the community, both in terms of those who are new to working in tech and in terms of those who specialize in other areas.

For these reasons, even if we are able to get back to in-person events quite soon, I have hopes that online tech events will continue to be a stronger presence than they were prior to the COVID-19 pandemic.

To do this, we need to solve some important problems about funding community and professional conferences, as well as improve the participant experience.

Problem 1: Sponsors Are Isolated Because Expo Halls Don’t Work Online

Most community and professional tech events, both free and paid, rely on sponsors to lower costs for attendees. Platforms for online events and staff to help organize and carry out the event cost real money, so this is still incredibly relevant for online events.

I’ve seen event after event attempt to replicate the ‘Vendor Expo Hall’ of an in-person tech conference. And at event after event, I have seen minimal to no engagement – I’ve yet to see or hear about ‘Vendor Booths’ working in an online format. After all, there’s no good online way to replicate the ways that events have made the Vendor Expo Hall more fun at physical events: free toys and swag, tasty food and drinks, games and magicians and entertainment which can take place right next to personal, real-world conversations.

Here are some things I’ve seen which didn’t work well. If you’re going to try them, change them up a bit to avoid known pitfalls:

Vendor happy hours at the very end of the day - an in-person happy hour sponsored by a vendor with refreshments served works well sometimes at physical events – often at these, attendees prefer to relax with others rather than return to an empty hotel room or sit in traffic during a busy commute time. At virtual events, everyone has screen fatigue, there’s no refreshments, and only one person can talk at a time. Plus, attendees would probably rather be spending the evening with their families.
- Idea: work more of the events suggested below into the ‘daytime’ conference schedule itself.
Vendor lunchtime hours when you provide competing entertainment - It’s great to provide lunchtime entertainment. But if you’re not otherwise working sponsors into your event schedule, don’t ask them to compete with paid entertainment at lunch.
- Idea: bring them into the lunch entertainment in some ways (and also work them into the event schedule in other ways).
“Visit a vendor booth” scavenger hunts - I’ve seen some conferences do simple code scavenger hunts to try to keep the sponsorship “exhibit” area in their platform from being a dead zone. I’ve yet to see or hear about this working, unfortunately. Some attendees snag the codes and leave, many ignore it altogether.
- Idea: make the contests more about meaningful interactions. More on this below.

Problem 2: Conference Participants Have a Hard Time Making Meaningful Connections

I’ve been part of conferences which have worked on building engagement in quite a few ways, which is great – but none have been great at helping participants create meaningful connections with speakers, each other, and sponsors. Here’s what I’ve seen tried:

“Chat roulette” style platform features: Several online conference platforms have decided that attendees want to experience the most stressful parts of online dating in a technical conference, and offer people the chance to be paired up with a random stranger for a few minutes on a video call. I haven’t seen this work well: attendees don’t seem tempted to join, and the platforms often seem to be error prone and not great at managing this.
- Idea: I have some suggestions for regular small-group gatherings and interactive workshops below – and there are ways to work sponsors into this as well!
Slack/Discord chat accompanyments: Several online conferences have offered (or attendees have created) social spaces where people can chat along with the event. While I think these can be a great back-channel for some and these can fill some needs for Q&A, for many these are overwhelming. They also become insanely busy very quickly if the conference drives a lot of people to them. So while they fill some needs, they don’t help most people feel like they made meaningful connections.
- Idea: keep evolving this model, but don’t rely on it for meaningful engagement.
Presenter Q&A during a ’live replay’ of a pre-recorded session: I personally really like this model as a presenter – it’s nice to know my session is safely pre-recorded and to be present to live chat with attendees in a text box. However, for many attendees it is hard to follow both a live chat window and a pre-recorded session that goes on simultaneously! While one may get a question answered, one leaves the session feeling a bit frazzled – rather the opposite of having a meaningful connection.
- Idea: Continue leveraging pre-recorded sessions if that helps you ensure a more consistent experience for your attendees, but allow time in your schedule for live Q&A after the session with the presenter, ideally with a webcam. They can recap chat that happened in the session, answer new questions, and reflect on how their session connects with other sessions at the conference.
- Another lesson learned: for your speakers, it’s critical to set expectations very early (such as in the call for sessions itself) about whether pre-recording will be needed, and what due dates and expectations you have for recording quality.

Problem 3: Conference Participants Suffer From Screen Exhaustion and Painfully Long Days

Most conferences have kept similar time frames and flow of sessions as they’ve moved into online events. However, the attendee experience is different: there’s no physical movement between sessions. There’s no change of scenery. And it’s often even harder for participants to disconnect fully from their work: they end up checking emails frantically during breaks, in early mornings, and in evenings.

Ideas: Mixing up “types” of sessions between more passive learning and interactive sessions will help a bit (but not do the whole job)

Experimenting with different “tempos” for online event days is important for conference organizers.
Experiment with more partial-day events rather than only full-day sessions. Could an online platform even allow you to have an event across more than one week?

Part of this is mindset for the organizers: resist the urge to believe that the selling point of your conference is a relentless bombardment of information across multiple tracks. Focus on maximizing the value participants will take away instead, which means helping them absorb information and have some time away from the conference screen.

Those are the three biggest problems I’ve seen.

Now for the fun part: ideas on how to make this more sustainable, more rewarding for participants, and more awesome!

Ten Ideas to Make Online Conferences More Awesome

These are numbered, but not ranked.

1. More Birds-of-a-Feather Chats, and Work In Your Sponsors - One thing I’ve seen that worked very well was mixing regular time periods into the conference schedule where conference attendees were invited to join live small-group discussions led by a moderator. Multiple discussions on different topics were live at each time. These were held in a format that allowed attendees to share webcams and microphones. Moderators came prepared with a list of questions and topics to spark the discussion.

This helps attendees have time for more meaningful engagements. The difference in format between this and a more passive learning session provides a needed change of pace. And you also have the option of inviting your vendors to participate in these, particularly the topics which are related to their mission and areas of interest. (This doesn’t necessarily mean doing demos or selling products, but rather being part of the discussion.)

2. Workshops, some Sponsored - I’m not going to lie, workshops ain’t easy to do well, whether in-person or online. But these are absolutely worth experimenting on, as when they are great they are amazing for your participants. Some notes:

Since this is hard, allow plenty of planning time, start with shorter workshops with fewer tech requirements, and do pilots first to find your groove. Build on your experience across subsequent events.
A key to a good workshop is scoping and selecting the right topics. Don’t fall into the trap of thinking tech workshops need to be all about how an engineer performs a technical task while they follow along in a cloud environment: that’s super hard mode! Start with “softer” topics around professional development or making strategies to effect change in the modern work environment.
I am personally a fan of the Lightning Decision Jam workshop format, which can work well online with access to resources like Mural . A great thing about this format is that it utilizes non-verbal communication effectively, and there’s a lot of folks who aren’t a fan of talking to a bunch of strangers in a video call. Find some formats you can suggest to your workshop organizers, but allow them to be creative.
For effective workshops, you need a way to limit the number of people in the workshops, which may require pre-registration to pull off. It also helps to have some volunteers available to switch workshops and balance out the groups if you have some which are low on numbers.

3. Commercial Contests - I know, especially for paid conferences, attendees may not love the idea of seeing sponsor commercials as part of the main conference track. But what if you gamified this for your sponsors, and added a contest for your attendees related to it? This could be really fun.

Keep the commercials short - 90 seconds to 2 minutes
Play the commercials live and reveal them throughout the conference, have an emcee announce them and reference the contest each time
Have multiple categories people can vote on – funniest, best production quality, most thought provoking, most educational, most charming, etc.
Open the voting at a specific time in the conference once they’ve all been launched, have a YouTube channel where attendees can watch them again in case they missed any
Encourage vendors to add a URL at the end of their video or to appear at the bottom of the screen the whole time for lead generation purposes

4. Prizes for Meaningful Engagement - As I mentioned above, I’ve seen simple scavenger hunts fail miserably. Just “visiting” a vendor booth in an online world doesn’t make for engagement with the vendor. Why not change the contest up? Your conference platform needs to provide some way of interacting with sponsors, whether it’s by chat room, video chat, or even email options. Ideas to hype throughout your conference to encourage engagement:

“Best question” contest: allow each sponsor to nominate up to N (whatever number is appropriate for your scale) people who asked the best questions for a prize draw. The winner draw can be done and announced at the end of the conference.
Sponsor trivia contest: create a trivia game where all the questions are about your sponsors and how their offerings relate to your attendees and their pain points. Source the trivia facts from your sponsors ahead of time, and if you’re doing a booth leverage things in their written materials and live demos. Have good prizes! Emphasize how valuable it is for the conference for contestants to share their email addresses and pain points with sponsors (but have a way for them to specify their contact preferences clearly, as noted below). This is a great “late in your day” session option for near the close of your event. Platforms like https://kahoot.it/ can make the trivia fun. (Note: do some test runs and allow for latency between the game and your viewers.)
Focus groups, with prizes for attending: Have time slots for 20-30 minute focus group interviews run by your sponsors. Give a small prize (gift cards, stickers sent by mail, etc.) to attendees to who participate in at least N number of focus groups.

5. Help Vendors Learn and Create Leads - Why would vendors want to sponsor community and professional events when they can run their own online events at a pretty low cost and gather all the leads they want? The answer is that vendors want to find new potential customers and to learn about where customers are going in the marketplace.

In other words, conferences can work to help vendors with both short term and longer term goals. Successful vendors understand that building their brand over time is important. They understand that learning about potential customers, their pain points, their perspectives, their goals, is key to their survival. Ideas:

Vendor surveys: Offer options for vendors to link attendees to relatively short (5 minute) surveys asking attendees about their pain points, interests, goals, etc. A contest for attendees who complete surveys is a potential encouragement for this one if you have interest from a good amount of vendors.
Vendor focus groups: See the “workshop” section above for more info on this. A focus group can feel more approachable / less intrusive for an attendee, as it’s clear that the point is to help the vendor by sharing experience and that the attendee may not necessarily be ready to buy a product.
Lead generation is still important. Don’t skip this! Do still work on building and improving the ways in which you encourage attendees to share their contact information with vendors when it’s beneficial for both parties. But add value around that wherever you can.
Showcase free training resources from sponsors which employees can use after the event: Many vendors have free online resources which can be very valuable for attendees (whether or not they are a customer of that vendor). This is everything from YouTube Channels to whitepapers to online courses. Ask your vendor if they have these, and showcase them for your conference participants in multiple ways throughout the conference – be it on banner adds, downloadable lists of resources, short video clips on the main stage, etc.

6. Ideas to Overcome “Awkward Chat Room” Experiences - I know quite a few folks who hate awkward silences in online meetings. It makes their skin crawl. I know others who feel very shy at speaking up in an online session. Others who can’t stand turning on webcams.

And then there’s the problem of having 12 people in a video room, and everyone starts talking at once, then apologizing. There’s That Person who can’t help dominating the discussion. There’s so many ways that things can get awkward.

And also, let’s face it, video calls get super boring real fast these days. Ideas to make it more interesting:

Provide non-verbal ways to participate: Apps like Poll Everywhere break up the monotony and allow people to give non-verbal feedback that is immediately visible to the group online (as long as the person hosting the polls/surveys are screen sharing).
- Other apps like Mural also provide ways to do this, but have fewer controls if you are worried about spammers or rogue feedback in more open events. (Poll Everywhere has some moderation abilities on free text style questions, and you can also stick to other question types if needed.)
Use Breakout Rooms, if you can: If you’ve got a larger group, apps like Zoom let you split folks into smaller breakout rooms for certain periods of time. This is useful for some workshop formats and focus groups.
Plan a game or activity: If you’re doing something like vendor chats or happy hours, I think a planned activity of some sort is really helpful. Relying on the guests who show up to provide the topics of conversation is often the cause of the awkwardness.

7. Pre-Recorded Sessions, Live Q&A After - Many conference organizers like to have some or all the sessions at an online event available in a recording ahead of time. This helps them ensure that the show will go on, no matter what technical problems ensue. (See my note above about the importance of mentioning your recording requirements when you do your call for speakers.)

I understand the appeal of pre-recorded sessions at a very personal level, having had my home internet stop working completely on the first day of a three day conference recently. I was working off a series of not-great-bandwidth cellular devices until it was repaired by line crews outside my home EXACTLY as the conference ended on the last day. My sessions were 100% saved by the fact that I’d pre-recorded them.

But as I mention above, it’s a huge challenge for attendees to multi-task and learn at the same time. I’ve yet to see a platform where a presenter could pause a pre-recorded session to handle a question live. Instead, chat and Q&A flows past while the session marches forward.

Part of the solution longer term involves explaining this challenge to conference platforms, and asking them for ideas to solve it. In the meantime, adjust your session timings accordingly!

If you want to do pre-recorded sessions in 40 minute timeslots, ask presenters to do 30 minute sessions and be available for the session and a live Q&A spot in the 10 minutes after the session.
Allow webcam in the live period if you can.
Encourage presenters to come prepared with a couple of things to discuss if Q&A is slow, such as how their session relates to others at the conference, how attendees can learn more, the most common misunderstandings on the topic, etc.

8. Shorter Days, Longer Events - I know event organizers are used to going flat out for a few days and then wrapping everything up. That makes sense when people need to travel to attend your event. But the game is very different with online events, and it’s time to change tempos.

In teaching online courses over the years prior to the pandemic, I learned that shorter sessions spread across more days helped attendees learn more:

They were able to take care of the most urgent things in their workplace in the hours that they weren’t in class
They were able to absorb things by spreading learning out over more time
They were able to literally “sleep on things” – several reported having woken up understanding something more clearly!

This is going to be a big mindset change for organizers in terms of marketing. Conference organizers are used to advertising on being the “biggest” conference with the most concurrent sessions. But with this approach, attendees leave your event feeling burnt out and strangely lonely. Will that bring them back?

A key part of improving attendee experience in online conferences will be experimentation with both session formats and conference timing. Think about how to help your attendees have the best experiences with new scheduling patterns.

9. Plan Carefully for Multi-Timezone Events - Timezones are hard, y’all. They’re hard when planning an event, and they’re hard when attending an event.

If you are an event organizer and are evaluating an online platform, a critical thing to evaluate is how the platform presents times to people who may be watching from different timezones. How easy is it for people to understand when sessions begin and end? It needs to be super easy.

Assume that the group of people who enjoy time zone math does not at all overlap with the group of people attending your event.

Depending on what technology you have to work with, you may not have a “perfect” option as an organizer. But do your best to try to provide whatever workarounds you can to make your attendees from different time zones feel at home. And express how important this is to the people who make virtual event platforms, please. This is a courtesy your guests will notice!

10. Be Clear – But Not Paranoid or Greedy – About Vendor Sessions - This is a funny one, because it’s more of an issue in some tech circles than in others. In both DevOps and Privacy themed conferences, it’s fairly normal for vendors to give sessions which feature how their products work. The sessions are typically clearly labeled so you know what you’re going to see. It’s not a big deal.

This is a good model for others to embrace. Offer sponsoring vendors the chance to give sessions explaining how their solutions give value. Clearly label the sessions so that attendees know what to expect and can choose accordingly.

This model actually works better in an online conference format than it does in an in-person one. Your sponsors will be highly aware of how easy it is to quietly and quickly leave an online session which is dull or generally unhelpful! There is none of the awkwardness of stepping through a row of people in a physical room to keep people there. The online format means that vendors need to bring relevance and value in these sessions, which is great for both attendees and conference organizers.

This Is Just the Beginning

This is only ten ideas. There are many more! What we need is creativity.

I’d love to see tweets, blog posts, and LinkedIn articles about what you think will make online conferences more sustainable and better experiences.

Coping with the Pandemic (T-SQL Tuesday #132)

help@KendraLittle.com (Kendra Little) — Tue, 10 Nov 2020 00:00:00 +0000

This post is part of the monthly TSQLTuesday blog event. This month’s topic is from Taiob Ali.

Taiob asks how are we coping amid the pandemic.

I Have Two Thoughts Overall About How Well I’m Coping

Thought 1: Wow this is rough.

Thought 2: I am incredibly lucky and privileged, and it’s still rough.

My Pandemic Story Started With an Intercontinental Move

At the beginning of March 2020, I moved with my husband and two dogs from Portland, Oregon to Cambridge, England. The virus was in the news at that point, but we had no idea how it would spread and how the world would change.

After nearly ten years of working at home, my intention in moving was that I’d begin to work from the Cambridge office for my employer, Redgate. I’d also periodically travel to visit customers in the UK and Europe.

Things didn’t work out exactly as we’d planned, of course: I worked in the Redgate office for exactly two weeks before the UK went into the first lockdown. I’ve been working from home ever since – at first from a small rental house (which we’d thought we wouldn’t spend all that much time in awake, haha) and now from a more comfortable home which we were very lucky to find and move to after the first lockdown let up.

Looking Back, I Was Lonely as Hell for the First Few Months

I was very happy to have my job and my husband and dogs with me, but I felt very homesick, stressed, and incredibly lonely throughout spring and summer.

But almost everyone was lonely. When I talked to friends at home and mentioned I was homesick, some mentioned that they also felt homesick– and they hadn’t gone anywhere. Everywhere, we were out of place, and worried about safety and health.

To soothe my stress and worry, I relied on habits that don’t scale well: eating sweets, making cocktails, not moving much, and not setting many goals for myself. I was grateful to have a job and that my family was OK, even if they were far away. But I didn’t want to work on myself.

6+ Months Into the COVID-19 Pandemic, I’ve Made Some Changes

Over time, I began feeling more and more sluggish, down, and anxious. I realized that I could work on good habits and help myself feel better, even while the pandemic continued.

In early September I began working on eating better. “Eating better” looks radically different for different folks – for me, from experience the key to eating better is to largely eliminate processed sugar from my diet. If I avoid processed sugar most of the time, I begin enjoying vegetables more and generally make better decisions about what to eat, when to eat, and how much to eat.

I also largely stopped drinking alcohol.

I know. This post is a drag. WHAT IS A PANDEMIC WITHOUT ALCOHOL?

Well, for me, it’s a pandemic where I make better daily decisions about my health. I haven’t stopped drinking entirely – I think it’s OK for me to occasionally have a glass of wine when there’s cause for celebration. But “occasionally” is once every two weeks or less often. I try to avoid alcohol altogether if I’m feeling stressed or sad.

With my cycling coach, Freyja.

Two weeks ago, I got an indoor cycle for the house and have started up daily spin workouts. I’m also challenging myself to do a bit of beginner yoga and to hold a plank pose once a day for however long I can manage.

What’s Made the Biggest Difference?

Taking on the habit of exercise as a positive coping mechanism has been a massive help. I’m finding that physical exertion is helping me to be less anxious even in stressful times. I also quite like that I can set goals around my fitness, track them, and work to improve.

And I also must say that my job with Redgate and my wonderful coworkers have made a huge difference. Not only is having the security of a regular paycheck a massive blessing, but working with understanding, open, and caring people is something I’m truly grateful for. My colleagues have helped me get through this weird, strange time.

Do I Have Any Tips?

My main tip is that it’s never too late to start working on a good habit.

I didn’t work on building good habits during the first part of this pandemic. But I eventually came to a time when I was ready and able to begin doing that. I don’t win anything by feeling bad about the first part – there’s no point in that. I may as well celebrate what I’ve been able to do, and work on motivating myself to keep going.

Could I Have Done This With Less Privilege?

This is a hard one, y’all. Not everyone is lucky enough to have a job, much less one with supportive colleagues. Not everyone has a family who are healthy and able. Not everyone has the time to obtain and cook fresh foods for dinner. Not everyone can afford an indoor cycle. Not everyone has a safe place to take walks outside. Not everyone has their health.

It’s an incredibly tough time. And if all you can do is get by each day, I certainly don’t want to imply that you are doing it wrong. You’re not. We each find our own way through this life the best that we can.

Also, I’ve shared a bit about anxiety before, and to be clear I don’t think that simply “good habits” are a cure-all for anxiety and depression in general. Right now these are what is helpful to me, but I’ve been hugely helped by medical professionals before when that was what I needed.

If you’re suffering significantly from anxiety and depression, I encourage you to please reach out for help and be open to seeking help from a doctor. Please know that you do not need to suffer alone.

Down Tools Week 2020 (10 minute video/podcast episode)

help@KendraLittle.com (Kendra Little) — Thu, 06 Aug 2020 00:00:00 +0000

It’s Down Tools Week at Redgate.

About This Episode

In this episode, I share what “Down Tools Week” is, what I’m working on this week, and why I think it’s a terrific experience for fostering creativity, innovation, and teamwork. In closing I share some thoughts on variations of this kind of project which you might use in your own organization.

🔥 LISTEN: Prefer to listen on the go? Get this episode on iTunes, play the audio file, or find Dear SQL DBA in your favorite podcast app.

Common Antipatterns in Database Development (28 minute video/podcast episode)

help@KendraLittle.com (Kendra Little) — Thu, 30 Jul 2020 00:00:00 +0000

Many people use entrenched processes for database development that have been in place so long that it’s hard to imagine doing it any other way. In this episode, I share three things that should NOT be normal for database development– but which are incredibly common.

The Episode

This episode is inspired by the book, “The Unicorn Project”, by Gene Kim.

🔥 LISTEN: Prefer to listen? Get this episode on iTunes, play the audio, or find Dear SQL DBA in your favorite podcast app.

Everything I Never Wanted to Know About Collation (14 minute video/podcast episode)

help@KendraLittle.com (Kendra Little) — Thu, 16 Jul 2020 00:00:00 +0000

I’m not an expert on collation in SQL Server, but there are a few important facts which I’ve had to learn the hard way. In this episode I share the basics, along with helpful resources to learn more.

Localization Correction

The default collation on a SQL Server install will be “the oldest available version that’s associated with each specific locale.” So if your Windows installation has a different localization than mine – I always use English (USA) – then you’ll see a different default collation than the one I discuss in the episode. The main point is still the same: it won’t be the recommended collation for *new* development.

Resources

Questions About SQL Server Collations You Were Too Shy to Ask, by Robert Sheldon - https://www.red-gate.com/simple-talk/sql/sql-development/questions-sql-server-collations-shy-ask
Collation and Unicode Support - https://docs.microsoft.com/en-us/sql/relational-databases/collations/collation-and-unicode-support

🔥 LISTEN: Prefer to listen on the go? Get this episode on iTunes, play the audio file, or find Dear SQL DBA in your favorite podcast app.

How to Spell and Capitalize tempdb for SQL Server

help@KendraLittle.com (Kendra Little) — Wed, 15 Jul 2020 00:00:00 +0000

No matter what I write in this post, some of y’all are going to tell me I’m wrong. That’s inevitable, because I’m writing about tempdb: a database so complex and mysterious in SQL Server, that even the spelling and capitalization of the database name is a topic of great disagreement.

Note: Nearly Everyone Has Been Inconsistent About This

You might think, “This is easy, just check Microsoft’s documentation to find the answer.”

That’s a great idea, except you’ll find multiple variations of “tempdb” on Microsoft.com:

This docs page uses tempdb (capitalized as “TempDB” in the link text)
This docs page uses tempdb (lowercase even when first word in the title)
This support page uses tempdb (But “Tempdb” when title case is needed) – most Microsoft support pages seem to follow this pattern
Bob Dorr, one of my favorite Microsofties who writes on SQL Server, likes to use all caps in his post
Pam Lahoud, another of my favorite Microsoft experts on SQL Server, uses tempdb (lowercase) in her post

So the good news is, if you’re writing for casual usage, there are a lot of different alternate spellings out there, and that’s OK. Most people do NOT use a space and make “Temp DB” two words, but you’ll see variations of that around the internet as well.

However, it can be nice to have a sense of the “most accepted” current spelling and capitalization of a term. This is especially useful if you’re building a tool with a GUI and you want it to look as “standard” as possible, or if you’re writing for a company website.

Here’s my take on what’s won the contentious “tempdb” spelling and capitalization wars.

My Recommendation Is: tempdb, All Lowercase, but Title Cased as Tempdb When Required

A few years back, I believe I saw Paul Randal point out on Twitter that tempdb is a real database, and the database is spelled tempdb. This simple argument has persuaded quite a few of us who write in the SQL Server world that there’s no use for any spaces in there, and that the “db” bit shouldn’t be capitalized. (Time has passed and I can’t find the tweet, so hopefully I’m attributing this correctly.)

You will find some variation in capitalization of tempdb on the SQL Skills blog, but Paul tends to use lowercase “tempdb” even at the beginning of a bullet point.

My colleague Kathi Kellenberger is the editor of Simple Talk. On posts she edits, she uses lower case tempdb, but there are exceptions as in when tempdb is the first word in an article title. (This is a good article to look at as a style guide, in my opinion.)

I have also noticed that other frequent writers in the Microsoft Data Platform space have standardized on “tempdb” somewhat, so if you choose this you are in good company:

Paul White, who has written more deep technical content on tempdb than anyone else I know, uses tempdb
Erik Darling uses “tempdb” in this new post on SQL Server 2019’s in-memory features for tempdb
Andy Mallon uses “tempdb” as he reminds you to stop shrinking your tempdb
As mentioned above, recent Microsoft support pages have largely standardized on this style

I try to be consistent using “tempdb” on this site, and I mostly succeed at this lately. I tend to be massively inconsistent about title casing in general, but I do my best. (Truth: I didn’t even capitalize the title of this post like I usually do, I had to go back and edit it after publishing, haha.)

Thanks to Piers for the Suggestion on This Post

My colleague Piers is on the SQL Monitor team at Redgate. The team was chatting about how tempdb is so complicated that it’s not even obvious how to spell or capitalize it, and he said, “you know, maybe someone should write a blog post.” 💡

And there’s always room for one more post about tempdb, right?

The Manager Guide to Git Training for DBAs - Video and Podcast

help@KendraLittle.com (Kendra Little) — Thu, 09 Jul 2020 00:00:00 +0000

Learning Git can be daunting for DBAs. In this 20 minute episode, I discuss why learning a VCS is necessary for DBAs, then give three tips on scoping your project, choosing the right tools, and making sure the project is successful.

Resources

The cheat sheet I mention in the episode is at /2019/11/27/my-git-cli-cheat-sheet/
A written version of this content is over at https://www.red-gate.com/blog/the-managers-guide-to-git-training-for-database-administrators
Here’s the full episode list of Dear SQL DBA

Thoughts as Pride Month 2020 Comes to a Close

help@KendraLittle.com (Kendra Little) — Tue, 30 Jun 2020 00:00:00 +0000

I’ve begun working on developing a couple of small habits this month, thanks largely to Andy Mallon’s helpful advocacy.

My pronouns are “she/her.”

I only became aware of why it’s helpful to share your pronouns in the last year. Before that, I hadn’t given much thought to it. Andy gives a great explanation of why this is helpful if this is something you are comfortable sharing – here is one quote:

For some folks, the obvious pronouns aren’t their preferred pronouns. You share your pronouns to help normalize it so that everyone feels more comfortable sharing pronouns.

Andy Mallon in Hi, I’m Andy. My pronouns are he/him.

I’ve started to edit my various biographies online to include my preferred pronouns. I plan to request that conference organizers include an optional space for pronouns in their slide deck templates as well when I speak at events. This is also an easy thing to include in my Slack profiles and other interactive tools.

Championing Anti-Harassment Policies and Codes of Conduct Instead of Being a Jerk About Them

My inner teenager isn’t so fond of things like dress codes – and for some years I thought of a code of conduct as being pretty similar to a dress code: unnecessary in the modern world. I was very wrong.

I now recognize that harassment and discrimination happens frequently, and that it can easily happen in any group. Harassment and discrimination often impact those who are marginalized in a group the most. However, even those who are well known in a group may have a hard time speaking up when they are victimized or when they see a peer being victimized. Anti-harassment policies help people understand that they can and should speak up, they explain who to approach about the situation, and they also give a framework for the event handlers to follow when working through the situation. I now see that this is very useful and needed.

I also now recognize that anti-harassment policies are important for expressing community values of inclusion. It’s a good thing to talk about what we find important as a group, and to express what works for a community, and what doesn’t work.

This is even good business sense.

“The smartest thing for people who are running conventions is to recognize that in both the short and long run, it’s going to be better for your convention if everybody knows they’re going to be treated with respect,” he said. “If you don’t do that, the younger people, the people who are vital to your field, are no longer going to feel like your convention or conference is a welcome place, and they will create spaces that are more welcome to them. You don’t want to be the ones who are left behind.”

John Scalzi quoted in Why Your Meeting Needs a Harassment Policy

My commitment now is to asking if an event has an anti-harassment policy before I commit to speaking at it, and to also check and make sure that the policy is inclusive of…

everyone regardless of gender, gender identity and expression, age, sexual orientation, mental or physical ability, physical appearance, body size, race, ethnicity, religion (or lack thereof), technology choices.

Andy Mallon in What’s Important in a Code of Conduct?

If it isn’t possible for the event to commit to this inclusion, it may not be the best place for me to spend my time right then. If it is possible for the event to do this, then I should send them positive feedback and help them promote this as part of a healthy community.

These are small things, but I can do them. The more of us who do them, the easier it gets.

New Free Course: The Dirty Secrets of NOLOCK

help@KendraLittle.com (Kendra Little) — Mon, 29 Jun 2020 00:00:00 +0000

I’m excited to begin moving over courses from SQL Workbooks and making the material available here. The first course up for grabs is The Dirty Secrets of NOLOCK.

A Little Bit About the Course

What happens when you use NOLOCK hints in your code, or set your isolation level to READ UNCOMMITTED in SQL Server? In 50 minutes of online videos, you will learn what NOLOCK means, why NOLOCK can return incorrect results (and other problems), what allocation order scans are (and how to get them), and other risks and options for reading uncommitted data.

Course lessons include:

What is NOLOCK? (3 minutes)
The dirtiest secret: NOLOCK returns garbage data (10 minutes)
The fast alternative to NOLOCK: indexing (5 minutes)
Why NOLOCK is a misnomer (7 minutes)
Bad phenomena that can occur under NOLOCK (2 minutes)
Uses for NOLOCK: admin queries and garbage data (5 minutes)
Allocation order scans (13 minutes)
A common error with NOLOCK (6 minutes)

The Course Is Free, but Please Donate If You Find It Helpful

If you find my free courses helpful, show your appreciation by donating to one of the following organizations:

Black Girls Code – whose mission is to “introduce programming and technology to a new generation of coders, coders who will become builders of technological innovation and of their own futures.”
The Trevor Project – whose mission is to “end suicide among gay, lesbian, bisexual, transgender, queer & questioning young people.”
The NAACP Legal Defense Fund – whose mission is to “achieve racial justice, equality, and an inclusive society.”
The LGBTQ Victory Fund – which “works to change the face and voice of America’s politics and achieve equality for LGBTQ Americans by increasing the number of LGBTQ officials.”

How to Rename the Master Branch to Main in Git in Azure DevOps

help@KendraLittle.com (Kendra Little) — Fri, 26 Jun 2020 00:00:00 +0000

I believe that language matters, and that it is worth our effort to move away from language associated with slavery and racism whenever possible.

The Process

Azure DevOps doesn’t technically allow you to rename branches, but you can work around the issue by creating a new branch from master, setting the new branch as the default branch, and deleting the master branch. Here’s the Microsoft documentation on this topic.

In this video, I create a new default branch named main, then update a pipeline dependent on the default branch name.

Note for Other Platforms

🔥 NOTE: If you're using Git in something else like GitHub, you can do this in a simpler way.

Create a Build with YAML for SQL Change Automation in Azure DevOps

help@KendraLittle.com (Kendra Little) — Thu, 25 Jun 2020 00:00:00 +0000

I used to make fun of YAML because I was scared of it. I still make fun of YAML, but I’m not scared of it anymore now that Rob Sewell showed me how to avoid having to write it myself.

Video Tutorial

In this 11 minute video, I show how to set up a YAML build pipeline in Azure DevOps. My build is configured using a Local Agent, and will be building a database using Redgate’s SQL Change Automation.

Use Chocolatey to Install Multiple SQL Server 2019 Instances for Testing

help@KendraLittle.com (Kendra Little) — Fri, 05 Jun 2020 00:00:00 +0000

I’m working on a project where it’s useful to automate environment setup and teardown for testing some devops deployment scenarios for databases using transactional replication.

Tools I’m Using

To make this easier, I’m using:

Octopus Deploy Runbooks to organize a series of commands (they have a free tier, FYI)
Chocolatey to install multiple SQL Server 2019 Developer Edition instances (I’ve written about choco before, it’s also free)

The Challenge

While I’m not at all a fan of “stacking” multiple SQL Server Instances into one Windows Installation in production, it’s fine for this testing scenario.

But I came across a little puzzle when trying to get this to work: when I tried to use choco install or choco upgrade for my second instance, it saw that SQL Server 2019 Developer Edition was already installed and did nothing. There is a –force option which can be used to install additional instances, but the SQL Server Installer will throw an error if you use –force and pass it information for an instance which is already installed. I needed to add a little extra PowerShell to make my script re-runnable and simple move on gracefully if an instance is already installed.

The Solution

Here is the code pattern I finally landed on for each instance installation in my runbook:

$inst=(get-itemproperty 'HKLM:\SOFTWARE\Microsoft\Microsoft SQL Server').InstalledInstances

if ($inst.Contains('SQL1')) {'SQL1 installed, no action taken'}
else {
choco install sql-server-2019 -Y --force --params="'/SQLSYSADMINACCOUNTS:domainname\username /SECURITYMODE:SQL /SAPWD:MyNotVerySecurePassword /IgnorePendingReboot /INSTANCENAME:SQL1 /INSTANCEDIR:c:\MSSQL\SQL1'"
}

Thanks to this StackOverflow answer for helping me along.

I’m looking forward to sharing more of my progress on this project as I go!

Learn T-SQL for Free Online: Starting Today

help@KendraLittle.com (Kendra Little) — Mon, 11 May 2020 00:00:00 +0000

Thanks to the support of Redgate, I’ve launched a new course which teaches you the basics of T-SQL. The course is totally free, no logins required: we don’t even ask for the email address.

Check Out the Course on Redgate University

The course is here:

https://www.red-gate.com/hub/university/courses/t-sql/tsql-for-beginners

Each week has an embedded video for the course, along with a link to the syllabus and scripts. The videos also have a timeline in case you wish to jump to a particular part of the discussion.

Want to Join Live?

I’m publishing a new module each week, and we have five weeks to go. There’s a calendar reminder in the course if you want to join me live on Wednesdays at 3 pm BST / 11 AM EST, or you can catch the videos each week and follow along.

Want to Learn Query Tuning, Reporting Services, or Power BI?

Redgate’s Community Circle initiative has other free courses on these topics. Check out the free course list here.

Learner's Guide to SQL Server Query Tuning

help@KendraLittle.com (Kendra Little) — Fri, 01 May 2020 00:00:00 +0000

Following on from my Learner’s Guide to SQL Server Performance Triage, I’m tackling Query Tuning. In this guide, I’m experimenting with an outline style rather than expanding each paragraph.

Why Do We Need Query Tuning?

We’ve seen incredible improvements over the last 15 years:

Storage speed has massively increased due to advances in both storage technology and network bandwidth, CPUs have become much faster, and prices for memory have dropped dramatically.
Database optimizers are constantly improving and finding ways to make queries more adaptable and to conquer areas of poor optimization.

Yet there are still professionals who make a good living tuning queries, and training others to tune queries. This process involves finding specific slow queries that are key to the performance of an application and making strategic changes, whether in the code, the database structures, the instance configuration, or something else, to ensure that these specific queries consistently execute with a given speed or to a required standard of performance.

Why?

Data sizes are also dramatically increasing
Customer expectations about performance / speed of applications have also risen
Customers expect to not have to wait for results now – they want to see current status right away

Who Needs to Do Query Tuning? Who Doesn’t?

Query tuning is done by:

Database administrators
Developers who specialize in performance or in databases specifically

Full stack developers don’t generally do query tuning unless they have a specific interest or work experience. This is a specialization rather than a “quick learning” task, so most full-stack developers simply don’t have time, and they need to engage a more specialized person to help. Teams who don’t have a readily available specialist may periodically bring in consultants to help with this.

There are also many database administrators who manage databases where only “basic” availability and performance are required. These databases are used by cost-conscious organizations who don’t need every database to be tuned like a race-car: most of their databases are used by internal users who are used to moderate performance.

What Skills Are Involved in Query Tuning?

I am not a wizard at TSQL, but I still became pretty good at query tuning. A very unscientific estimate of what skills make someone good at query tuning:

65% - the time, interest, and resources (including a network of people to ask) to build an extensive map of “performance trivia” – this is patterns that don’t optimize well, edge conditions where performance goes bad, understanding of trace flags and configuration on TSQL.
20% - interest and ability to learn how the database engine optimizes and processes queries using indexes and other resources, and what impacts concurrency as multiple queries run against a live database at the same time
15% - understanding of the TSQL language and different ways of rewriting a query to produce a given result set

Popular Built-In Query Tuning Tools and How They Work Together

Execution plans

These are a “map” of how the query is run behind the scenes.

“Estimated” execution plans show the choices the optimizer has made to run the query, including how many rows it estimates will flow through different parts of the plan
“Actual” execution plans are estimated plans updated with runtime statistics for things like how many rows flowed through the plan. If the plan is “adaptive” it will contain some information about which options were chosen.

Wait Statistics

Was the query slow because it took a long time to read a lot of data? Or because it couldn’t get a lock resource? Wait statistics help sort this out.
Some wait statistics are included in actual execution plans as of SQL Server 2016 SP1+, but not all waits are tracked per query due to overhead

Query Store

SQL Server 2016+, all editions

This feature tracks execution plans and aggregate runtime metrics (duration, cpu usage) along with aggregate wait stats
This also has the ability to “freeze” plans
Query Store information restores with the database itself, so it can be shared between environments if desired.

Dynamic Management Views and Performance Counters

These help understand overall instance bottlenecks during the slow performance
Example: overall wait statistics for the instance and metrics about storage latency during the time the queries performed poorly can help explain if the query really needs to be tuned
- Query tuning frequently needs to do callbacks to “workload tuning”

SQL Trace and Extended Events Traces

These are tricky to use for query tuning because it’s easy to slow down your workload and cause performance problems when tracing:
- Execution plans (filtering doesn’t help in this case, the plans are all examined / collected and the filter is applied too late)
- Wait statistics (filtering can help here, but the data collected is so massive that you have to be very careful – and sorting through and querying the collected data is also quite cumbersome

Tough Problems

Parameter Sniffing

If a query is “sometimes fast, sometimes slow”, this may be the cause
For parameterized queries, SQL Server “sniffs” the values supplied for the parameters on first execution. This plan is reused when other values are provided on subsequent execution unless something happens to cause recompilation.
An example…
- The CustomerOrderDetails procedure is initially compiled for @CompanyId= 1001, a tiny customer with one order. A plan expecting a very small amount of orders is generated, which allocates very little memory for sorts and joins.
- The CustomerOrders procedure is then run for @CompanyId= 128, our biggest customer with a million rows of details about their orders. As the query runs all the estimates are too small, there are inefficient one-by-one reads and memory spills are happening all over the place
- Someone restarts the SQL Server because “it’s slow”. This causes queries to all compile freshly. CustomerOrderDetails runs for the first time with @CompanyId = 128 and compiles a plan suited to many rows, and runs fast. Nobody understands what happened.
  - In this case the “slow plan” would not longer be in the DMVs. It would be stored in Query Store if that was in use.

Contention

It’s difficult to predict how queries will interact with one another in a live workload

Shared resources:
- Query workspace memory – a certain amount of memory needs to be allocated for sorts/ joins/ moving data around in a query. Lots of queries executing at once which need significant workspace memory can cause a problem with this.
  - Sometimes the queries are estimating they need far more of this memory than they need and it will need to be tuned – people are unlikely to realize this outside of a live workload
- The number of queries doing modification and the approach to locking is difficult to predict outside of a live workload
  - Changes in query plans can cause blocking when it wasn’t present before
Changes in server resources – even improvements – can cause blocking when it wasn’t present before
- Example: moving to a new server with more memory and faster CPUs saw an increase in lock waits, because storage waits went down and query execution sped up.

Is Testing in Production Required?

Often, yes. One of the examples of this is parallelism.

Tuning the level of parallelism for a workload, and for specific queries in that workload tends to be quite hardware specific, and you need a live environment.

Workload “replays” are available within SQL Server’s toolkit but they are:

Time consuming to set up
Only replays – you can’t “amp up” the activity meaningfully (deleting the same rows 10 times isn’t the same as deleting different rows 10 times)

Automated Query Tuning: History and Evolution

Automated Plan Correction

SQL Server 2017, Enterprise Edition

Built on top of Query Store
Detects queries that are sometimes fast and sometimes slow
Can recommend changes only if desired
Can freeze plans, test if it helps, and react accordingly
- Freezing is intended as a temporary fix – it’s recommended that a user evaluate the query for tuning as a longer term fix
Very good feature for helping identify parameter sniffing

Intelligent Query Processing

IQP features have been releasing over the last few versions of SQL Server
Several of these features address common query tuning issues in SQL Server
Examples:
- Scalar function inlining: Scalar functions have historically been very poorly optimized in SQL Server, causing slow row-by-row performance
  - This feature “inlines” the functions into the larger execution plans and rewrites the plan accordingly
- Adaptive joins: Adaptive logic is built into plans in some cases so that not only a “single path” for execution is available in the plan
  - This currently applies to limited scenarios
- Batch mode execution:
  - Optimizes how SQL Server handles scans of data by processing multiple values at once
  - This originally was available only for queries which reference columnstore indexes.
  - Batch mode for rowstore indexes introduced in SQL Server 2019

Common Mistakes and Pitfalls in Query Tuning

Lack of connection between DBAs and Development teams

Over-reliance on traces

Lack of knowledge of execution plans in the team

Overuse of hints

Lack of understanding of SQL Server isolation levels and “optimistic” options

Join the Best Worst Code Contest Today at 5pm BST / Noon Eastern #BestWorstCode

help@KendraLittle.com (Kendra Little) — Fri, 24 Apr 2020 00:00:00 +0000

I have an idea! Let’s have a “Worst Code Contest.”

Why Do This?

Bad code is really fun: at least, when you write it on purpose. And other people’s bad code is even more fun.

But seriously, I’ve needed to write a lot of bad code over the years when creating demos and reproducing various scenarios. It uses your brain in an unusual way, so it’s a fun thing, but it also teaches you a lot!

How to Play

The game starts at 5pm BST / Noon Eastern / 9 AM Pacific today. I’ll be livestreaming the contest here, but you can play whether or not you join the stream. Here’s how it works:

I will post a challenge on Twitter to write your best worst code that either does a specific thing in TSQL or produces a specific result set when run against a SQL Server
- I will use the tag #BestWorstCode
- I will push a “challenge file” describing the challenge to https://github.com/LitKnd/BestWorstCode/
There will be a 10 minute time limit for each challenge.
- You can use whatever resources you want to put your solution together, but you must submit your solution within 10 minutes to compete. The solution needs to be in a set of short scripts (it can’t require cloning a whole repo or restoring a database – the most beautiful awful code is succinct)
Tweet back a link to your BEST WORST sample code using the tag #BestWorstCode.
- You can save your code anywhere that works, as long as a link works and I can copy it to a clipboard and test it. One easy way to do this is to create a free GitHub account and create a Gist for each sample. But blog posts are fine too, or whatever you like using that is easy to link to.

We’ll do TWO challenges, so the contest will probably take 45 minutes altogether, with time for chatting through different submissions.

You can join for one or both of the challenges. If this is fun, we can do it again sometime soon, too!

What Do I Need to Have Installed to Compete?

You need three free things:

A SQL Server (don’t use a production instance, LOL, Developer Edition is free)
A copy of the pubs sample database (script to create it is here, this is a Microsoft script shared under the MIT License)
A dream

And truthfully, the dream is optional.

Do I Have to Play to Join the Livestream?

Nope. Join the livestream if you wanna chat while you play along, or just chat. You can also play without joining the livestream. It’s all good.

Is There a Prize?

You will win the admiration of two corgis and a sense of superiority that will last you all weekend long.

Let’s Write Some Awful Code Together!

We can make something terrible and beautiful at the same time.

Update: The Recording

Make Database Code Reusable in SQL Source Control with Deployment Filters

help@KendraLittle.com (Kendra Little) — Wed, 08 Apr 2020 00:00:00 +0000

When I first began working with databases, I was lucky to land a job at a little start-up which had solid development and operations processes: all our code, including database code, was in version control. We had a strong database architect who instilled good coding practices into the way we managed database code as well: code was expected to be reusable whenever possible.

For example, there was one set of code which managed error handling and logging for all our databases. It lived in one repository in version control, and was deployed into many databases. This code resided in its own schema, in this case the “EVT” schema. Similarly, there were other bits of “reusable” database code for managing configuration, transferring data between instances, documenting objects and procedures, and other common tasks.

There Are Huge Benefits to Establishing Reusable Pieces of Code for Your Databases

The top five benefits I can think of, are…

It becomes easy to instill good practices, such as quality error handling, across different teams, without each team needing to learn in-depth about the topic. They simply need to learn how to call the reusable code.
It is easy to test if core good practices have been implemented, by checking for the existence of the reusable code and whether it is working as part of deployment processes
It is far more efficient to version and maintain code commonly used across databases by storing it in one repository and creating tests for that code itself
Those doing production operations gain a consistent way to quickly gather information from databases in production across every environment. For example, 15 years later I still remember that I would run the EVT.LogList procedure to check for error output in databases when diagnosing an issue
This vastly simplifies writing triage documentation for your databases

But How Do You Implement This with Modern Tooling?

Back in the day, our company wrote our own tooling to help us with this, because there simply weren’t any vendors who provided this functionality. Writing and maintaining that tooling required a significant investment on our part.

These days, there are lots more tooling options available (and now I work for a vendor in this area, Redgate Software). However, it can be puzzling to determine how to implement reusable code in modern database tooling, especially “state-based” tooling such as Microsoft SSDT or Redgate SQL Source Control.

That’s because when you do a state-based deployment, tooling of this type compares the object definitions which you have stored in source control with the target database, and generates commands to make the target database look exactly like the objects you have in source control.

A diff occurring between a committed version of database code and a target database. In a state-based deployment, tooling automatically generates the code to be deployed from this differencing task. Without the use of filters or special tooling, any objects not in version control will be scripted to be dropped.

This becomes tricky if you are only deploying part of a database, especially if there are any dependencies which exist between the part of the database you are deploying and the rest of the database. If you do nothing, the state-based solution will assume that you want to make the target database look exactly like what is in version control and only like what is in version control, so it will script DROP commands for anything else.

Deployment Filters Simplify Managing Reusable Database Code

How you solve this problem depends on your tooling. Since I work for Redgate, I naturally invest my time in working with our tooling the most – so from here on out I’ll be talking about deploying SQL Source Control projects with SQL Change Automation, which has features which help with this problem.

SQL Change Automation has graphic extensions/plugins as well as PowerShell cmdlets. I’ll refer to the PowerShell cmdlets when describing how to do this, but know that the very same functionality is available in the plugins:

If your reusable code is in a repository all by itself, you can use the -IgnoreAdditional option to deploy everything from your build package and simply ignore (aka don’t drop) additional objects in the target database
If your reusable code is in a shared repository (perhaps with other reusable code, perhaps with something else), and you want to build the whole thing but only deploy part of the code (say, a schema), this can be done by specifying a filter file saved from SQL Compare using -FilterPath when you create the release artifact.

In other words, no matter how you are versioning the code you want to make reusable, there is a path towards being able to safely deploy this code in an automated fashion to target databases as needed.

Reusable Code at Work

There are other interesting patterns which deployment filter files enable. For example, Chris Unwin and I recently worked on a case where we were asked how to make schema based deployments to a large number of production databases as efficient as possible. This can be done with a variety of orchestrators – in this example the use case is to have Octopus Deploy manage the release orchestration.

We’ve just published a 46 minute YouTube video, “Reusable Schema Deployments with SQL Source Control and Octopus Deploy” which gives you a view of this at work. (To be clear, Chris 100% did all the hard work here and I’m riding his coattails!) If you’d like to check out individual parts of this video, a table of content with links is at the bottom of this post.

00:40 Overview of the problem we are solving: finding the most efficient way to deploy individual schemas (which are each stored in version control once) to a large number of production databases (which each may contain a mix of some or all of the schemas)
3:00 A discussion of state and migrations approaches, and why we are discussing a state-based solution with SQL Source Control for this problem
5:10 An overview of the two parts of the solution: Release Artifacts are reusable, and deployment filters effectively “ignore” anything that is excluded by the filter (it will not script “drop” commands for them)
8:05 An overview of the solution workflow
10:32 Demo begins! An overview of the databases used for the prototype environment proving this out
14:00 An overview of the build pipeline in Azure DevOps Services which builds SQL Source Control and hooks things up with Octopus Deploy – and a brief excursion where Chris de-mystifies YAML and shows Kendra that dealing with it in Azure DevOps is MUCH easier than she thought
22:00 A view of how the prototype environment works in Octopus Deploy (including a view of the lovely aquatic dark theme in Octopus now)
24:00 A view of the procedural deployment steps in Octopus Deploy using Redgate SQL Change Automation and an explanation of how steps can be configured to run against certain environments
28:00 A view of the filters created for schema deployment, and a demo of how to create them in SQL Compare
33:15 A view of how managing “environments” as groups of databases makes it easy to perform deployments of a schema to many databases with a minimum of steps in Octopus Deploy
35:02 Time to show it at work! Chris kicks off a deployment
36:25 A review of the Release Artifacts created, and the changes and code therein
40:39 Summary /recap of the benefits of this approach

Connecting to the SQL Server Community Online: Uplifting Folks

help@KendraLittle.com (Kendra Little) — Mon, 23 Mar 2020 00:00:00 +0000

We’ve got a session coming up in the free Redgate Streamed virtual conference on, “How database DevOps levels up remote work.” When thinking about that session I began thinking about other ways to improve remote work.

While I worked remotely for nearly ten years until recently and have established practices which work well for me , moving to another country and the general chaos and uncertainty right now as the world tries to cope with a global pandemic have increased my feelings of worry and loneliness.

I have found that certain folks in my Twitter feed help me feel connected, combat my worry, and also add some good old fashioned nerdiness to my day. They level up my remote work day!

If you could use a few more connections in your life and would like to get started on Twitter, I highly recommend following these friendly folks who love working with data, and who bring thoughtfulness and happiness into the community:

Cathrine Wilhelmsen, aka @cathrinew

Cathrine loves teaching and learning, and her tweets always teach me something, give me something to think about, or improve my day in some way.

Cathrine’s bio: “SQL/Data geek, Microsoft Data Platform MVP, speaker, blogger and chronic volunteer. Taller on Twitter! Loves sci-fi, chocolate, coffee, and cats. She/Her.”

Mala Mahadevan, aka @sqlmal

Mala constantly learns and shares. She is one of the kindest people I know as well as an honest person, and her tweets remind me that one can confront things and be kind at the same time.

Mala’s bio: " SQL Server Enthusiast, PASS Regional Mentor-NE region, here to learn and follow all things SQL. My views do not reflect views of my employer. She/Her"

Drew Furgiuele, aka @Pittfurg

Drew loves dogs, databases, and people who are doing their best.

Drew’s bio: “KE8SQL. Former Data Platform MVP turned Microsoft PFE. Co-host of http://doingtheirbest.com podcast. Tweets and opinions my own. The toughest guy in Letterkenny”

Jess Pomfret, aka @jpomfret

Jess is stellar at automation. Her twitter feed also always makes me feel like doing something active, even if it’s just a quick set of exercises in my home office (and as a lazy person I really need this kind of inspiration)

Jess’ bio: “She/Her. SQL Server, PowerShell, Crossfit, Proper Football”

Andy Mallon, aka @AMtwo

Andy provides a lot of dog photos and asks essential questions in polls, like “What’s the appropriate thing to say when you toot?” These are the things that keep me sane. He’s really good at managing databases too and he probably even posts about that sometimes.

Andy’s bio: " Boston Terror. Database Artist. Dog Dad. Carbatarian. I am a four letter word. Vaccinated. he/him. #Food #Politics #SQL #Activism #TeamOxfordComma"

Steve Jones, aka @way0utwest

Steve lives on a ranch, so he’s got horse photos, ranch photos, and mountain views. He’s also got a really grounded-but-positive perspective on things. He’s thoughtful. He has good ideas. (Proof: he not only founded SQLServerCentral, he also co-founded SQLSaturday events back in the day!)

Steve’s bio is truly minimalist. It is: “Editor, http://SQLServerCentral.com”

Buck Woody, aka @BuckWoodyMSFT

Buck is more than a person. He’s more of an experience. Follow Buck for learning resources, tutorials on loads of things across the data platform and analytics, terrible puns that you’ll still laugh at, and also a lot of kindness.

Buck’s bio is empty.

There Are Loads of Awesome People in the Community, and This Isn’t an Exhaustive List: Not by Far!

These are simply the people I thought of first today! If I didn’t list you, it doesn’t mean I don’t love ya :)

Outlining My Session: Index Tuning in an Agile World

help@KendraLittle.com (Kendra Little) — Fri, 20 Mar 2020 00:00:00 +0000

I’m giving a session on index tuning at the upcoming Redgate Streamed free online conference. The conference will be held April 1-3 2020, register here and join us!

I love talking about index tuning and I know this area well, so I’m excited to put together this new session. I thought it would be fun to share my process of outlining and creating the session, leading up to the event.

The session title and abstract I wrote this week is…

Index Tuning in an Agile World

Software developers increasingly work across databases and applications, and there are fewer and fewer dedicated database developers each year. This produces a new opportunity for database administrators and developers who are interested in performance: the role of the incremental index tuner. In this session you will learn a framework for developing and tuning indexes in SQL Server, and how to apply this framework in an Agile development context.

Who Is My Audience? What Is My Goal?

Whenever I’m writing or rehearsing a session, I remind myself to think about who my ideal audience is. In this case it’s fairly broad: it’s developers or DBAs who care about performance of their code, but they aren’t yet experts at index tuning. The interesting thing right now is that often the people in this audience don’t have all of the information they need to do index tuning by themselves: from the DBA side, they typically have access to the production system but often don’t have access to the codebase, or the design of the queries. From the developer side, they often have the opposite situation: they can get to the code and change it if needed, but they don’t have regular access to production.

My goal in this session is to get all of these people started by showing an example of index tuning, and proposing a workflow in which a performance problem can be identified, a change to indexes can be suggested and reviewed, and the change can be agreed upon and deployed through the environments needed to reach production.

My Initial Outline

I like to begin putting together sessions with an outline of how I think the session will flow. This outline may change entirely over the course of writing the session, but it feels right to start with it. Here’s what I’m thinking we can cover in an hour:

Intro: The Problem Space

Disconnected teams: developers lack production access, DBAs lack access to code and ability to change it
Both groups often are unfamiliar with index tuning

Learning by Example: The Goal Workflow

Show an example query with erratic performance
Demo: How can the problem be discovered by a DBA with production access?
Demo: How can the problem be discovered by a developer without production access?
Example: testing different indexes to improve performance in a development environment
Example: using work items and a pull request workflow to propose index changes
Gotchas: where can this go wrong?

How to Improve Your Workflow

What if your database code isn’t in version control yet? How do you get started?
What if other folks don’t want to change / want to keep things they way they are?
What about designing indexes from the ground up instead of tuning an existing system?
References / links for learning Git, branching / merging strategies
References for index tuning basics and further learning

I’ll be working on this session more next week, so stay tuned to see how it comes together.

I’m grateful that I get to work with others in the community in this topsy-turvy time, and that I have you as a reader. Stay safe, take good care of yourself and your families, and I hope that you are healthy and able to join us o n April 1-3 2020, for Redgate Streamed.

Announcing Redgate Streamed, April 1-3 2020

help@KendraLittle.com (Kendra Little) — Thu, 19 Mar 2020 00:00:00 +0000

We’re thrilled to announce Redgate Streamed: a 3 day online virtual community conference, to be held April 1-3, 2020. Register at Redgate.com/RedgateStreamed

About the Event

In this 9 minute livestream, I give you a rundown of the event schedule and tells you why I’m excited about each session.

Support COVID-19 Response

For every registration that attends Redgate streamed, live or on-demand, we’ll donate $1 to the World Health Organization’s COVID-19 response fund (in addition to a separate donation we’ll be making. Even if you can’t join us live, be sure to register and you’ll receive the recordings to watch on-demand, plus you’ll help us raise our gift level!

The Learner's Guide to SQL Server Performance Triage

help@KendraLittle.com (Kendra Little) — Thu, 19 Mar 2020 00:00:00 +0000

I’m introducing a series of “learner’s guides”: overviews of a given topic, chock full of links and references. For this first post, the information is based on what I learned when I was part of the team at Brent Ozar Unlimited who put together the original First Responder Kit and built a related consulting practice using those tools.

What Is Performance Triage, and When Do We Do It?

Performance triage is done when a production database is suspected to be the cause of slowness or of loss of availability. Many times when perf triage is done it’s unclear what the source of the problem is, and the goal is to either identify what is happening in the database or instance to cause this problem, or to rule out the database as much as possible.

In other words, most applications aren’t written with enough built in observability to clearly tell if the problem is in the application layer, the database layer, the network or somewhere else. But the database is often an early suspect in an investigation.

Who Does Performance Triage?

Database administrators (DBAs) have been the main people to do performance triage over the last 20 years. This is largely due to separation of duties: DBAs have typically been responsible for the performance and availability of production databases, while access has been limited to these environments for developers. Often DBAs are part of an IT group which is managed as a separate business unit from that of software development. DBAs also often are responsible for databases developed externally from the company, such as vendor databases.

In recent years, responsibility for production environments has been shifting, at least when it comes to in-house development. As organizations work to be able to deliver new features to customers more quickly in competitive environments, movements such as Agile development, DevOps, and Infrastructure as Code have been shifting roles: it is more common than ever for developers to need to maintain responsibility for their code throughout the entire deployment process, and for Database Administrators to be asked to participate in the development process earlier to help with quality design and testing.

Monitoring tooling also better enables communication to both of these teams, and automation is used to help manage permissions for different environments, allowing access to be granted quickly, but only when it is needed and approve.

“DMVs” Revolutionized the Way Performance Triage Is Done

Prior to the release of SQL Server 2005, “Profiling” using SQL Trace was the most widely used methodology, most often run by the SQL Server Profiler tool.

SQL Server 2005 changed methodologies dramatically with the introduction of Dynamic Management Views and functions, which are commonly referred to as “DMVs”. DMVs hold a rich amount of information regarding the present and recent aggregate performance of a SQL Server instance and its databases. The major benefits of DMVs include:

Much lighter overhead than tracing
Data on everything from internal performance of the SQL Server to which user queries have been run and how often

However, many DMVs in SQL Server are cleared when different events occur, such as an instance restarting, a database going offline, a query recompiling, a configuration changing, or memory pressure. For this reason, monitoring tools are very popular and are used to harvest information from DMVs (as well as the sources listed below), store them in a separate repository, and use them to provide historic performance information.

Sources for Information in Performance Triage

In addition to dynamic management views, monitoring tools and folks performing manual triage often reference:

Operating system error logs. Windows is most commonly used to run SQL Server in production environments, so this is usually the Windows Event log, both System and Application logs
Windows Cluster Manager logs if Failover Clusters and / or Availability Groups are in use
SQL Server Error Log - SQL Server maintains its own log of errors
Virtualization metrics (if applicable)
Performance counters - SQL Server performance counters can be accessed via Perfmon or queried via SQL Server in sys.dm_os_performance_counters
- There is some overlap between performance counters and DMVs - occasionally perf counters are either easier to use or hold information not available or easily attainable from DMVs
- OS performance counters are available via Perfmon and are often relevant as well
SQL Trace (either Server Side SQL Trace or live SQL Trace via SQL Profiler) and Extended Events (XE) traces - these have more impact on the SQL Server and it’s easier to cause a performance problem with them, but they can be useful when handled with care
Query execution plans and related query runtime metrics - execution plans may be harvested via DMVs, Query Store, or a trace (warning: these are expensive to collect in traces, even using Extended Events)

When Is Performance Triage Difficult?

Performance triage is most difficult when monitoring of the SQL Server either isn’t in place, or monitoring is only being done by a high level “platform” style monitoring system such as System Center Operations Manager (SCOM). This is because a significant amount of critical information about performance is cleared when failovers happen, restarts occur, configuration is changed, or memory pressure occurs.

While “platform” tools sometimes do have specialized management packs to attempt to gather some deeper information, in practice most teams find it difficult to harvest and use this information. For this reason, it’s quite common for teams to use both a platform level tool for high level monitoring across many types of databases, applications, and services, as well as a specialized SQL Server monitoring tool for monitoring databases and instances, both on-prem and in the cloud.

While it is possible to do performance triage in a reactive model and begin gathering information after the problem happens, in this mode you are left waiting for the problem to happen again before you can begin to diagnose it.

If you’d like to see an example of SQL Server specialized monitoring running against a live system, Redgate’s SQL Monitor has an online demo which runs against environments including the live SQL Server Central databases as well as cloud PAAS workloads. (Disclaimer: I work for Redgate.)

SQL Server Performance Triage Checklist

A Frequent Check: What’s Happening Now?

Whenever I begin triage, I often start with checking: what queries are running RIGHT NOW? No matter what monitoring system or tooling is in place I like doing this check with the free utility sp_WhoIsActive. I like this because sp_WhoIsActive is a stored procedure you may run against the instance, and like many operations people I like to see information in multiple tools – so a monitoring tool can help confirm what I see in sp_WhoIsActive and vice-versa.

Throughout triage I typically stop and run sp_WhoIsActive multiple times to confirm that the instance is responding quickly and that I don’t see signs that something is going wrong. After all, you never want to make a problem worse by triaging it, and you don’t want to be oblivious to what is going on while looking at logs of past behavior.

Phase 1: Look for Big Gotchas

When triaging performance I generally want to start big picture and focus on the following sources:

Windows Error Log - How many times did I forget to check this at the beginning, only to realize there was an obvious error which was either the source of the problem or a major immediate clue? At least 10 times. Do a quick pass on this early on!
Key virtualization performance counters or checks - Using a hypervisor? Check (or ask the managing team to check) if the balloon driver kicked in when you had the problem, the VM was migrated between hosts, or something else unusual occurred.
SQL Server Error Log - Have there been stack dumps? Did the instance restart? Are there messages about super slow storage latency? Again, this is always worth a quick pass early on as it can save you loads of time.
Do we have a ridiculous lack of resources? - Spending extensive time troubleshooting performance on a 1 vCPU instance or one with 4GB of memory is going to cost way more operationally than is worth it.
I like to use the free sp_Blitz script from Brent Ozar Unlimited (note: I’m biased, I worked there!) to quickly check for the following. Just be sure to get used to the script so you don’t get lost in the details and can find the high level info quickly when you run it. (Again, I like to cross-check this information in a monitoring tool – I’m all about triangulation and multiple tools, though.)
- Bad settings which can harm performance – super low max memory, auto-close or auto-shrink on databases
- Signs of “poison wait stats” which can kill performance (I think I may be the person who coined that term! It me!)
- Signs of super-slow storage from storage wait stats DMVs like sys.dm_io_virtual_file_stats - this DMV stores an average since startup, so don’t jump to conclusions too fast, but sometimes this holds the answer
- Database backups / RPO and RTO - Although database backups / recovery models are unlikely to be the cause of your performance or availability problem, please do take the time to glance at these: because one problem is that often backups get cancelled when performance is bad in order to lighten load on the system. This can leave you exposed to MAJOR RISKS. You don’t want to be tweaking configuration and skipping backups. Well, you just don’t want to be skipping backups.
- Major missing indexes - SQL Server records when it thinks a nonclustered disk-based rowstore index will be useful to speed up a query in its DMVs and execution plans. Don’t assume that something in this list is the reason poor performance happens, but possibly this will inform your list of “suspects”

Phase 2: Narrow Down: Is It the Infrastructure, Is It the Workload, Is It One Query?

If I haven’t found a clear direction to go in so far and instead I’ve just ruled out the big gotchas, now I start narrowing in. Hopefully I have either a specific time period that I’m looking at where I know the problem occurred, OR I know it’s an ongoing problem which I can observe now. When it comes to anything that isn’t constant, having a specialized monitoring tool is critical.

If I don’t have a specialized SQL Server Monitoring tool, I need to start sampling and capturing information without trying to put too much pressure on the system. This can be done, but you can’t do everything and once and you have to be careful – so you must do a dance of watching what is running, sampling information, analyzing it, and then moving on to more samples. Data I sample or review in a monitoring tool includes things like:

SQL Server wait statistics for the impacted time - sys.dm_os_wait_stats and sys.dm_os_waiting_tasks
- These are an invaluable tool to finding out why SQL Server is slow, but it’s quite an art to use them. You need to know what to ignore, which waits are available in different versions of SQL Server and what they mean, and which waits are more problematic than others.
- It’s incredibly useful to have a baseline of wait statistics, either collected manually or by your monitoring system. Erin Stellato gives a background on wait stats and explains why here.
Storage waits / samples of sys.dm_io_virtual_file_stats and / or performance counter information for Physical Disk (Avg Disk sec/read and write, Disk Reads/sec and Writes/sec) for the impacted time
- How much are/were we using storage when there is a problem?
- How much storage latency do we have when there was a problem? (What latency is acceptable? As Paul Randal says, everyone has their own idea of what that is – here is his take.)
Blocking and/or deadlock reports, if available
- If you see lock waits in the wait statistics, blocking may have been part of the issue
- If you have a monitoring tool, it will help you analyze the chain of blocking and view deadlock graphs to decode which queries are involved
Query durations and runtimes in the problem period, along with query execution plans
- Both monitoring tool repositories and Query Store are rich sources of information for this
- If you don’t have either of those, you can reference sys.dm_exec_query_stats and related DMVs, but will need to play a guessing game about which of your suspects were running in the problem period, what their duration was then, and what execution plans they had

Common Mistakes and Pitfalls

The biggest mistake that I’ve seen with this is that folks rely too much on tracing. Not only can tracing slow down the SQL Server, but it slows down your triage process:

You have to wait for the problem to happen again
Traces generate a lot of data which are time consuming to sort through and analyze
You have to be cautious about the amount of things you collect to avoid causing a whole new performance problem, so it’s not uncommon to find that you weren’t tracing the right things to troubleshoot the issue after it occurs, and to need to start all over again

My most common personal mistake is that I’ve forgotten to check for errors in the OS and SQL Server Logs early, and missed clear indications of the source of the problem, wasting loads of time. This is easy to do in a stressful situation even when you have practice, so I recommend having a checklist or runbook for your team to use when triaging.

Another common issue is that not all SQL Server performance counters are useful, and some have persistent myths and misinformation to be aware of.

I’ve found that one of the hardest problems to identify is parameter sniffing. It’s good to be familiar with this in advance if you need to troubleshoot performance in SQL Server. A great resource on this is Erland Sommarskog’s Slow in the Application, Fast in SSMS.

How Is This Evolving, and What Does the Future Look Like?

One of the most exciting pieces of tooling introduced into SQL Server regarding performance tuning recently is Automatic Plan Correction. This requires the use of Query Store and it helps identify when queries are sometimes fast and sometimes slow. In other words it can help with identifying that tricky problem of parameter sniffing.

You may use this in an advisory mode, or let it test freezing different plans for you in production
Note that “corrections” are intended as temporary fixes: for long term performance, queries that are “corrected” likely need either query tuning and/or index tuning

Automatic Index Tuning is available in Azure PAAS editions

I’m not crazy about trusting SQL Server’s missing index suggestions – I look at them but take them as a suggestion to look at the related queries directly and do my own analysis, as they’re often oversimplified.
I do NOT trust ye olde Database Tuning Advisor.
But automatic index tuning is an interesting feature and it involves an element of testing.
I haven’t personally looked into this much, but I’m curious about it, and I can see it working well on the right workloads, probably workloads with relatively small data sizes.
Automatic index tuning does present interesting challenges in terms of how teams want to manage schema in source control and what indexes they want to deploy to a new workload if using a single-tenant-per-database architecture.

APIs for Customization from Monitoring tools

“Did that deployment/maintenance/configuration change we did yesterday cause a problem? When exactly did it happen, anyway?” Figuring that out has always been a toughie.
One of the cool things that SQL Monitor has done is to add an API for annotations so that teams can track significant events on the graph of performance metrics.
This can be automated to happen right away when an event occurs, or it can be used by team members to note times of interest, perhaps when triaging performance and noting times when a problem occurred.
As DBAs merge more into platform teams and/or combine with development teams, the use of APIs to both manage alerts and the ability customize how alerts are delivered will only increase, along with other custom features enabling teams to customize their responses to incidents. (Again, I work for Redgate, so I’m biased on my choice of tooling and examples.)

Why Re-teaming Is Good

help@KendraLittle.com (Kendra Little) — Fri, 13 Mar 2020 00:00:00 +0000

🔥 NOTE: This post is based on an editorial I originally published over at SQL Server Central.

Years ago, I worked on a fabulous team of eight database administrators

We supported more than a hundred developers who worked in an agile fashion. When I first joined the DBA team, we had a shared on-call rotation, but each DBA specialized in a certain area of the environment and regularly met with the development teams working on that area. DBA specializations rarely shifted.

Over Time, This Created Significant Stressors

Covering on-call for some areas became quite difficult in times of rapid change. It became impossible for someone to be out of the office for an extended period without delaying releases. When we got to the point that we had to decide whether or not to escalate a critical issue to someone who was attending a funeral, it was clearly time for a change.

To Gain Flexibility, the DBA Team Decided to Regularly Rotate Positions

Not everyone was thrilled. We each felt attached to the environment we supported: like a garden we had faithfully tended for years, it was hard to let it go! This made the transition a bit tough, especially at first.

Over time, we found that regularly rotating specializations had many benefits for our group. The most wonderful improvement was that it made it possible for team members to fully disconnect while they were off work, as other team members not only had deep technical familiarity with an area, but also our developers were used to partnering with more of us and everyone could adapt more easily.

Fast Forward to Today: Now I Work at Redgate

When I joined Redgate I was intrigued to learn that our Product Development teams run a self-selective re-teaming process across the entire division. Re-teaming just took place at Redgate in January 2020 for the second year in a row. Because it is at a larger scale than eight people, Redgate’s re-teaming model gives people the option to have a say on whether it is time for a change or not (rather than only enforcing rotation).

Self-selecting reteaming is used at Redgate for multiple reasons, which I’ve shamelessly stolen from an internal post written by Redgate’s own Chris Smith to share with you. Reteaming….

Realigns team size and purpose to reflect the larger organizational strategy for the year
Gives engineers autonomy based on the principle that people will be most engaged and motivated if they have agency regarding what they work on
Spreads knowledge, best practices, and innovation throughout teams, as people bring techniques, skills, and insights along with them from their prior work
Enables personal development
Reflects the recommendations of thought leaders, who champion this collaborative approach as an increasingly common characteristic of high-performing software development organizations.

Re-teaming Encourages Good Documentation Habits

This week I realized something obvious: if you’re a developer who has been working on a productivity tool like SQL Prompt and you move to working on a tool related to version control and automation, there are certain things you wouldn’t necessarily know about your customers. For example, most customers in financial services and healthcare can’t or won’t use SQL Authentication. There are many things like this.

However, this isn’t a blocker to re-teaming. Instead, realizing this helped me understand that I can help improve our customer profiles to make these types of requirements and preferences more clear to everyone.

Reteaming helps surface “tribal knowledge” and help make it clearer for everyone.

Although Change Can Be Tough, I Believe That We’ll Increasingly Find Reteaming Practices Popping Up in Engineering Teams Everywhere

And in my experience, it’s a good thing.

If you’d like to learn more about Redgate’s use of self-selecting reteaming, check out the “Dynamic Reteaming (with Heidi Helfand)” episode of the Ingeniously Simple Podcast.

Altering an Indexed View in SQL Server Drops All Indexes

help@KendraLittle.com (Kendra Little) — Mon, 02 Mar 2020 00:00:00 +0000

I learned an interesting thing about ALTER VIEW behavior in SQL Server when applied to indexed views. This is covered in the product documentation, but it’s not something I would have expected.

ALTER VIEW can be applied to indexed views; however, ALTER VIEW unconditionally drops all indexes on the view.

ALTER VIEW documentation

When I first heard about this behavior, it sounded like a bug to me: why should an alteration like adding a column to a view remove all the indexes?

A colleague of mine at Redgate wondered: is the behavior the same with the new CREATE OR ALTER syntax as it is with just plain ALTER? (Spoiler: I tested and it is the same: CREATE OR ALTER also drops all indexes on the view.)

One note: if you are using Redgate tooling to commit your database objects to version control, never fear: it handles this for you and will automatically put the indexes back on your indexed views if you alter them! This is true both for SQL Source Control and SQL Change Automation. Just be aware that behind the scenes the indexes must effectively be recreated to implement this change, so if large tables are involved in the view definition, deployment time will be impacted.

This Behavior Makes Some Sense When I Think About How Indexed Views Are Implemented

When you create an index on a view (we’re talking views using classic disk-based tables here), it materializes the view – in other words, it stores the data as if it is a table. That data is stored in rows on 8KB pages. These pages can fill up. So if you add a column to a view, just like adding a column which requires a size-of-data operation to a table, the database engine needs to rewrite those pages to add the new column to each and every row (depending on datatypes and nullability, etc etc).

Also, it’s quite possible that when altering an indexed view, the alteration makes the view no longer adhere to the many rules required for indexed views.

After thinking about this, I can understand why if you allow ALTER VIEW, the simplest way to implement that against an indexed view is to make it the user’s problem to ensure that indexes can be re-created after altering a view.

However, I Personally Wonder If ALTER VIEW Should Even Work for Indexed Views

This is one of those little niche behaviors that strikes me as problematic. From a user perspective, I’d personally rather that ALTER VIEW simply failed for indexed views and notified me that to change an indexed view, I need to drop the indexes, alter the view, then recreate the indexes which I need. Although that’s more work to me as a user, it helps me understand that this is not necessarily a cheap or low risk operation, and it reduces the risk of me accidentally deploying code that un-indexes a view which might be critical to performance.

However, this ship has sailed: the behavior is documented and established and unlikely to change in the SQL Server Engine anytime soon.

Want to See a Demo?

Check out this fantastic video by Erik Darling where he discusses and demos this behavior.

Come Work with Me! Be a Sales Engineer for Redgate

help@KendraLittle.com (Kendra Little) — Fri, 21 Feb 2020 00:00:00 +0000

Redgate is growing, and we have some fresh, new open positions around the world which would be a great fit for SQL Server developers or DBAs who would like to transition to a customer-facing role and develop expertise in Redgate’s solutions for Compliant Database DevOps. Here are the listings and locations for three of these roles:

As a DevOps Advocate, the Sales Engineering team is one of my favorite groups to work with at Redgate. These folks get to learn all about database DevOps, and they get to show clients the possibilities for how they can better develop and deliver database changes, as well as how to better protect their data in the software development process.

But the Sales Engineers aren’t only intelligent. Our Sales Engineering team is a group of kind, smart people who are just plain fun to spend time with. Whenever I’m on a call or a webinar with one of the SEs, we work as a team and we have a great time figuring out how to solve problems. Joining our Sales Engineering team means joining one of the most respected and well-known teams inside Redgate.

One Note About Sales at Redgate

Redgate practices ethical selling. Depending on your life experience, the words “ethical selling” may look a bit like an oxymoron. However, I’ve been working with our sales teams at Redgate for more than a year and I can attest that we really do work our hardest to make sure that we only recommend solutions for the client which fit the team’s needs and which will help the client succeed. There’s no “vapor ware” in this job, only helping people find a solution which empowers them to work more effectively as a team, be more productive, and deliver value for their organization more rapidly.

Don’t Know DevOps?

If you’re knowledgeable about databases and are interested in developing expertise in Database DevOps (but aren’t there yet), please feel free and empowered to apply! While it’s terrific to find an applicant who is well versed in it all, this is one of those “unicorn” jobs in my experience where strong experience in a few of the needed areas is what is needed, as there’s not exactly a huge pool of people who tick all the boxes.

If You’re Interested in the Job, Please Use One of the Links Above to Apply

If you’d like to specify me as the person who referred you on the application form, that’d be lovely – and full disclosure, I might get a bonus if you get hired. If you don’t feel like listing me, that’s OK too, we’d still love to hear from you.

I Can Do Better, We Can Do Better, Let's Do Better

help@KendraLittle.com (Kendra Little) — Thu, 13 Feb 2020 00:00:00 +0000

In the past week or so, the Microsoft Data Platform community has begun having a discussion about inclusivity, both on Twitter and across community blog posts.

This conversation began when a member of the community shared their story about being repeatedly mis-gendered and additionally feeling excluded, unwelcome, and hurt at a series of community events.

Historically the legacy SQL Server community has had quite visible support for Women in Technology, but has not shifted broadly/overall to a public focus on larger Diversity and Inclusion issues. There have been efforts to be more inclusive to people of color and members of the LGBTQ community, but these efforts have have not been nearly as visible or “popular” within the community as the efforts to support WIT.

I don’t write this in an accusatory way, only to describe the state of things as I see them. I have not worked to change this, and as a community member I share in the responsibility for the problem I see.

This Past Week I Have Seen More Care and Concern from Community Members Over This Issue Than I Expected

I’ve seen many comments from people saying that they want to learn to be more inclusive, and I’ve also seen folks asking for advice on how to speak and act in ways that will help make everyone feel welcome.

This pattern of asking how to do better is what makes me the happiest, because I do need help to learn to be inclusive by default, and to recognize when I am making a mistake without realizing it. I see that I’m not alone in recognizing this and I appreciate others who are making it a known pattern to ask for help in this way.

Here Is an Example I Thought About This Week from My Own Life

Years ago I began a job as a contractor (V-dash, holla!) at Microsoft. At the time there was a bit of an “outsider” feel to being a contractor, but I didn’t mind, I got to work on cool stuff, and I liked my fellow contractors just as well as I liked the full time employees.

One office over from where I sat was another contractor who happened to be transgender. We didn’t work on the same team, so we were never in the same meetings, but we passed each other in the hallway plenty. I never said “hi.” I never went out of my way to smile. I never said, “hey, do you want to grab lunch?” Looking back, there was plenty of evidence that my coworker felt unwelcome, isolated, and alone, and by doing nothing I simply added to that problem.

What I should have done was to be friendly and welcoming to my colleague, even though we weren’t on the same team. I should have introduced myself. I should have smiled in the hall, even if I’m always super awkward smiling at people I don’t know (insecurity, hi, it me). I should have asked if they liked what they were working on: like I did of plenty of other folks who were on other teams.

If I felt like I was a bit of an “outsider” because I was a contractor, I can hardly imagine how my colleague felt.

I did not realize this in any way until this week.

I’ve Learned a Lot This Week

I can’t go back and redo my past behavior, but I can do better now and in the future. I can ask others to help me recognize when I can improve. I can listen. And I think we as a community can do better as well. I am regretful that someone was hurt, but I am glad that we are having this conversation. Growth is usually not comfortable, but it is important.

Protect Your Prod Databases in Azure DevOps / TFS: Three Control Points

help@KendraLittle.com (Kendra Little) — Wed, 05 Feb 2020 00:00:00 +0000

When implementing any kind of automation for database deployments, it’s important to implement safeguards for your production environment. This is needed even in the best conditions when team members collaborate well and there is a high level of trust, for a very simple reason: accidents happen easily!

As a person who has accidentally destroyed quite a lot of things VERY RAPIDLY with automation😱, I am a fan of adding protection around production environments to lower risk for everyone, including myself.

On the other hand, it’s also desirable to empower team members to be able to quickly build and experiment with automation in other environments. It’s best to not lock down more than you need to.

In this post I’ll give a quick overview of three control points which help protect your production databases when working with Azure DevOps Server/Services (or legacy Team Foundation Services). The controls I’m discussing here apply to SQL Server instances where you are deploying changes with self-hosted agents.

There’s more than one way to do these things, so if you’ve got thoughts about your favorite alternatives, I’d love to hear about them in the comments.

1. Control Who Can Create New Pipelines Against Production Instances

While it’s useful to let folks create new pipelines and edit them against all sorts of development, test, or QA resources, not everyone should be able to create a pipeline, point it to an existing production database, and deploy changes to it.

One simple way to manage this is to create one or more agent pools for production SQL Server instances and use the security properties on the agent pool to limit who can create pipelines using that agent pool. For those who need to see the progress of pipelines/monitor deployments, there is a ‘Reader’ role available.

This could also be done with deployment groups for parallel execution.

2. Control What the Agent Can Do

When you install a self-hosted agent, you are prompted to configure not only which agent pool it uses, but also which account the agent will run under. If you wish for that agent to only be able to deploy to one SQL Server, you may choose to use an account which you have granted only permissions for that SQL Server instance.

In other words, if you are concerned about the agent being used to communicate with other SQL Server instances across the network in your production environment, you have the ability to control this by using an account which only has permissions granted against the SQL Server instance of your choice.

3. Control What Can Be Done Within an Existing Release Pipeline

Pipelines are quite flexible and there are many things you can do with them in terms of gating and approvals. Two very powerful controls which you may choose to configure right away for pipelines with production targets are…

Configure who can run vs edit vs view a pipeline: Pipeline security can be configured to allow people to interact with an existing pipeline in many different ways – for instance some users/groups perhaps should be able to execute the pipeline but not edit the pipeline definition.

Configure approvals within the pipeline: Stages may be edited to set approvals. An example with Redgate tooling is that you might create one stage to create a release artifact for a production change, then another stage which performs the deployment of that release artifact. You might choose to add an approval step at the entry to the deployment stage.

It’s OK to Start Slowly

If this is a bit overwhelming, remember that you don’t have to automate everything at once, and in fact it’s preferable to have a strong foundation for automation. The first step is to identify the current pain points and constraints for your team and start from there. If your database code isn’t yet in version control, beginning there is very likely the first step. If provisioning dev, test, or pre-production environments is slow and cumbersome, working on automating provision and deployments there will help increase your velocity. But when it does come time to begin automating deployments to production environments, do take the time to examine the available safeguards and implement the right controls for your organization.

High Availability Options for SQL Server 2008 in an Azure VM: A #sqlhelp Story

help@KendraLittle.com (Kendra Little) — Sat, 01 Feb 2020 00:00:00 +0000

I asked a question on Twitter yesterday: “If running SQL Server 2008 in an Azure VM (no SQL upgrade possible, extended support in place), is the best option for high availability Database Mirroring? Or something else I am not thinking of? #sqlhelp”

The Community Response

And check out the magic of the sqlhelp hash tag, I got loads of answers! So many that I’m actually selecting a few here for the purposes of keeping this post readable. First, John Morehouse , a consultant who does a lot of great work in the cloud and on-prem helped me figure out what would be needed for clustering (which isn’t something you can do simply with this version in an Azure VM)…

I belive you could also cluster it but you’ll need SIOS for the storage. I think. DisclaImer: I’ve never tried it. Mirroring should work as well and might be simpler. #sqlhelp

Microsoft’s Input

I also got input from two folks at Microsoft, Oscar Zamora and Amit Banerjee.

SIOS, mirroring or VM replication/Azure site recovery #sqlhelp

Additional Perspectives

Glenn Berry chimed in: he is also a consultant and specializes in topics like high availability and performance.

As others have said, Failover clustering using SIOS for the shared storage. Instance-level protection, that would be easier to maintain than DBM. #sqlhelp

Other folks responded as well. It can be so useful to get multiple perspectives on a topic, and I’m continuously amazed at how helpful folks in the Microsoft Data Platform community are when they can help out with a question.

Thanks to everyone who helped me out!

A First Look at the Redgate Hybrid Model for SQL Source Control and SQL Change Automation (Video)

help@KendraLittle.com (Kendra Little) — Sat, 25 Jan 2020 00:00:00 +0000

About This Video

In this 70 minute livestream recording, I kick the tires of a fresh new Azure DevOps demo environment showing Redgate’s Hybrid Model for SQL Source Control and SQL Change Automation.

Video

To skip watching me installing ZoomIt in the new demo environment, start at around 6 minutes.

Controlling Who Commits Code to a Given Database Schema in Git with Azure DevOps

help@KendraLittle.com (Kendra Little) — Sat, 18 Jan 2020 00:00:00 +0000

One of most the fun things about working as an Advocate at Redgate is getting to help clients determine their preferred workflow for database DevOps.

Teams often have unique requirements and are using different combinations of tooling, so figuring out the best way to accomplish what they need typically involves leveraging what I already know, collaborating with my coworkers and the client to generate ideas, researching and prototyping solutions, and then getting feedback from everyone.

Moving from Monolith to Microservices

One pattern which can enable this move is to at first increase the use of schemas within the existing large legacy database, and begin to document and understand dependencies between different parts of the codebase. Over time, the goal is to reduce dependencies and to begin to move portions of the database out from the large legacy database and into their own data stores. The schemas help clarify ‘ownership’ of code with minimal initial impact to performance and no need for initial data movement.

Limiting Who Can Commit to Each Schema

In other words, if team Fruitbats owns a schema which exists for the Fruitbats service, members of the Cheeseweasel team shouldn’t just go making changes to those objects. Instead, they need to talk to the Fruitbats team and discover what the best way is to accomplish what they need. At the same time, it’s valuable for team members to be able to see the code for all objects to support the existing legacy codebase.

I thought about a couple of different options in thinking through this, including things like using multiple projects, using a branching strategy, or figuring out some way to run checks in pre-commit hooks with Git.

Here’s What I Would Do

After consideration, the approach which I think is the simplest, but which has the most benefits, is to keep all objects in the database in a single project and use Pull Requests (PRs) with automatic reviewers to ensure the right changes are being made.

This is simple to set up with a combination of Azure DevOps and Redgate’s SQL Source Control tooling, because SQL Source Control uses an object’s schema name as part of the file name.

Here’s an overview of the setup:

A branch policy protects master and requires a PR to complete a merge
Automatic reviewers are set up
- Establish a user or group with ownership over each schema, and set up a rule so that if any files in that schema are changed in a PR, the group or user is added to the PR. It can’t pass until they review it (or someone with permission to override makes it so).
- Since SQL Source Control places the schema name at the beginning of file names, the filter for each schema can be as simple as something like: */fruitbats.*
Automation can further automatically run a build of code and run tests against the code when the PR is created, which can do things like ensure that new cross-schema dependencies are not introduced

I like this approach because while it ensures that the teams owning a schema will always be in the loop on a change, it is still flexible: there may be times when it is desirable to have a single person contribute changes to multiple schemas. This allows that without the need for permissions to be reconfigured, and it even gives a place for the owners of all those schemas to have visibility and input on the change early in the process.

In Other Words, This Was Simpler Than I Thought

The approach here can be a simple modification to an implementation of the Azure DevOps Release Flow branching strategy I wrote about last week: it’s a way to use feature branches with automatic reviewers. While there are certainly other ways to solve this problem, this workflow fits many teams and is also quite simple to implement and maintain.

Why I Like the Release Flow Branching Strategy with Git for Database DevOps

help@KendraLittle.com (Kendra Little) — Fri, 10 Jan 2020 00:00:00 +0000

When people begin applying DevOps principles to database development using Redgate tooling, often one of the first steps in the process involves getting database code into version control. Questions naturally come up about how to manage the flow of changes to database objects from development into production once changes begin occurring.

When using Git for version control, branches are a huge help when it comes to managing this flow.

But how do you use branches? It’s helpful to pick a strategy. There are many fine Git branching strategies out there, things like GitFlow and GitHub Flow and more – enough that it’s overwhelming to learn about these when you are just starting out.

The strategy that I recommend for folks who are starting out with database DevOps and Git is the Azure DevOps team Release Flow model with dedicated development databases. (Why dedicated development databases? Read more here.)

Why the Azure DevOps Release Flow Model Fits Databases Well

Topic / feature branches provide a place for experimentation
Pull requests provide communication, feedback, and review – a prime collaboration point for developers and DBAs for many teams
Code can be merged into the main code line without being deployed automatically (for databases, where rollbacks are often anything but straightforward, immediate deployment isn’t always grand)
Releasing at sprint milestones using a branch created for deployment often makes the most sense

Further, the release flow model can support both small and large teams.

I Drew a Diagram

Sometimes Other Models Make Sense

For instance, for those who have implemented a microservices model with robust automated testing in place, it may be more suitable to do something like a modification of the GitHub flow branching strategy.

But that’s not the case for most folks – most folks have databases which are more complex, with cross-database queries and downstream dependencies galore. For these more ordinary databases, the Azure DevOps Release Flow model is a really great place to begin.

Why I'm Learning Git via the Command Line Interface

help@KendraLittle.com (Kendra Little) — Fri, 03 Jan 2020 00:00:00 +0000

I’ve learned a bit about Git in the last year: I’m now quite comfortable creating and managing Git Repos in Azure DevOps. I frequently do demos with SQL Change Automation and SQL Source control with Git on Redgate’s YouTube channel, and I’ve published a Git Cheat Sheet for the Command Line Interface.

My Git Learning Journey

I love learning Git now: I’m very comfortable with a small set of core commands which empower me to get a lot done and I’m able to regularly add new bits of knowledge into my repertoire.

I was interested in learning Git before, and I tried and failed to get started learning more than once. Each time, part of the problem was that I tried to use graphical tooling which I thought would make the whole thing simpler, but which I found to be difficult to start using – and also prone to quickly getting me into complex situations which I didn’t understand.

A big differentiator to me finally learning Git was embracing the built-in command line interface (CLI).

Why the CLI Works Better

While command line interfaces are often harder to learn than graphical tools, in Git’s case I feel that it’s the opposite: the CLI helps you learn and focus. Here’s why:

You don’t need a lot: You mainly need to learn five or six simple, core operations which you will use 90% of the time – for everything else you’re generally going to use search engines and forum entries to figure it out case by case, anyway
It helps you learn: Mis-type a command? It will list commands spelled similarly.
Simple add-ins take away the toil: Most of the pains I’ve had with Git have had to do with forgetting my current branch context, or not wanting to type out a whole command – add-ins like posh-git quickly solve these problems and let me have a simple-but-easy experience

When I Use Graphical Tools

There is still a case when I’m happy to depart from the command line: when it comes to merge conflicts, I’m all about having a graphic tool to help me sort out the mess. I’m a long-time fan of the editor Sublime Text, so I’ve recently been happily working with Sublime Merge (which allows you to try it out for free).

But for everything else Git, I love learning with the CLI.

Resolving Merge Conflicts in SQL Source Control: The Basics (Video)

help@KendraLittle.com (Kendra Little) — Fri, 27 Dec 2019 00:00:00 +0000

In this 35 minute livestream recording, I commit conflicting code to a Git repo in Azure DevOps Services using Redgate’s SQL Source Control, then step through options to fix the conflict.

We first run through an example where we hit a conflict when pushing to the master branch and resolve that. Then we run through an example where we are using a feature branch and identify the conflict when doing a pull request to merge the change into master.

Resources to Follow Along

SQL Source Control is part of Redgate’s SQL Toolbelt. It is not free, but we have a free trial. If you don’t need automated build and deployments, you can also purchase SQL Source Control individually.
The Northwind database is shared by Microsoft under the MIT License on GitHub
I use the Sublime Merge editor in this demo to resolve the conflict. You can try that out for free at https://www.sublimemerge.com

Why You Should Take the 2020 State of Database DevOps Survey Today (Video)

help@KendraLittle.com (Kendra Little) — Thu, 19 Dec 2019 00:00:00 +0000

In this six minute video, I explain why you should take the 2020 State of Database DevOps Survey. The survey is open for a few more days – we’ve had a record number of responses, but we want YOUR input as well!

Take the Survey

Take the survey at Redgate.com/DevOpsSurvey.

Database Reliability Engineering (22 Minute Video)

help@KendraLittle.com (Kendra Little) — Wed, 18 Dec 2019 00:00:00 +0000

I was fascinated yesterday to come across the term, “Database Reliability Engineering,” which I hadn’t seen before.

In this 22 minute whiteboarding session, I talk about why we need new terms for “Database Administration,” and my initial understanding of what Database Reliability Engineering means by comparison.

Video

Online, Resumable, and WAIT_AT_LOW_PRIORITY Operations in SQL Server

help@KendraLittle.com (Kendra Little) — Tue, 10 Dec 2019 00:00:00 +0000

ONLINE operations in SQL Server were simple to understand for years – we got ONLINE index rebuilds in SQL Server 2005. That was it for a while.

Then, things got more complicated: we got more types of indexes. We got ONLINE options for schema changes that don’t involve indexes. We got more options for managing things like blocking, because online operations are really only mostly online: generally there’s going to be at least a short period where an exclusive lock is needed to update metadata. We now have some RESUMABLE operations coming in, too, for those big operations that are tough to handle.

Along the way, I fell behind. Because these features have steadily come out over a period of time, my brain simply didn’t register them all, or possibly I missed seeing them amid other announcements.

Having realized this, I did a little inventory: here’s my rundown of what I understand to be the current state of ONLINE options. Please feel free to add notes in the comments if I’ve missed things, and I’ll update the post accordingly. And I admit, in researching this post, I didn’t spend time looking at In-Memory OLTP tables … they come up so rarely that I didn’t go there.

ALTER TABLE ALTER COLUMN ONLINE (Disk-Based Rowstore)

Usage: most often changing nullability or data type. This is slow against large tables, because we’ve got to go rewrite all those rows and hold locks while we do it.

ONLINE options in Azure SQL DB, SQL Server 2016 +
Documentation lists the limitations and impacts. Search for “as applies to altering a column” to find it in the page.

🔥 INFO: There are no WAIT_AT_LOW_PRIORITY or RESUMABLE options for ALTER COLUMN ONLINE.

ALTER TABLE ADD COLUMN (Disk-Based Rowstore)

Usage: commonly done operation in routine software development when adding a new feature. This can be extremely slow when adding a non-nullable column with a default value, read about that in the link below.

There is an Enterprise Edition feature for adding a non-nullable column with a default value in SQL Server 2012+
Read about it from Remus Rusanu

🔥 INFO: This does not require or allow the use of ONLINE in the syntax. This is particularly tricky because it occurs in Developer Edition and doesn't occur in Standard Edition. 🤕

By the way, if you want to nerd out on what happens behind the scenes when you add or modify various types of columns, Remus has another excellent post on that topic.

CREATE INDEX ONLINE (Disk-Based Rowstore)

Usage: index creation is a common operation for both new features and performance tuning.

ONLINE options in Azure SQL DB, SQL Server 2005+ (Enterprise Edition required)
Documentation lists the limitations
RESUMABLE added in SQL Server 2019

🔥 INFO: There is no WAIT_AT_LOW_PRIORITY option for CREATE INDEX.

DROP INDEX ONLINE (Disk-Based Rowstore, Clustered Indexes Only)

Usage: I am very hesitant to recommend this. I am mentioning it here to explain why it is probably not your friend and your attempt to do something ONLINE could end up in a world of regret with a full transaction log.

ONLINE options in Azure SQL DB, SQL Server 2008+
From the documentation: “When a clustered index is dropped OFFLINE, only the upper levels of clustered indexes are removed; therefore, the operation is quite fast. When a clustered index is dropped ONLINE, SQL Server rebuilds the heap two times, once for step 1 and once for step 2.”
But wait, there’s more you should know: because nonclustered rowstore indexes use the clustering key as part of their own definition, this also causes rebuilds of all nonclustered indexes on the table – this can accidentally use a TON of space.
If you need to drop a clustered index, generally this pattern is the best bet:
- Drop or disable all nonclustered indexes (monitor for blocking, but a fast operation otherwise )
- Drop the clustered OFFLINE (monitor for blocking, but a fast operation otherwise)
- Create new clustered if desired (potentially with ONLINE option)
- Create or rebuild all nonclustered indexes (potentially with ONLINE option)

If you find a situation where you want to use this, I recommend testing very carefully with a production-like data set before unleashing it on your database.

ALTER TABLE DROP CONSTRAINT ONLINE (Disk-based rowstore)

Similar to the DROP INDEX for clustered indexes, this works with indexes created by constraints. Since Clustered Primary Keys are implemented with constraints, this basically allows you to do the clustered index drop mentioned above for a clustered PK.

CREATE COLUMNSTORE INDEX ONLINE

Usage: columnstore index creation is increasingly common, as mixed workloads / reporting off of transactional databases is in high demand and far more viable from a software and hardware standpoint than it used to be.

ONLINE option for Nonclustered Columnstore in Azure SQL Db, SQL Server 2017+
ONLINE option for Clustered Columnstore in Azure SQL Db, SQL Server 2019+
Documentation

🔥 INFO: There are no WAIT_AT_LOW_PRIORITY or RESUMABLE options for CREATE COLUMNSTORE INDEX.

ALTER INDEX ONLINE (Disk-Based Rowstore)

Usage: commonly used to defragment indexes during maintenance operations, sometimes used to adjust fill factor and other index settings

ONLINE options for rowstore in Azure SQL DB, SQL Server 2005+
Documentation
ALTER index is used for maintenance and for things like changing properties such as data compression, fill factor, and index padding
WAIT_AT_LOW_PRIORITY in SQL Server 2014+ (using it requires ONLINE=ON)
RESUMABLE = ON in SQL Server 2017
Single-partition ONLINE rebuild in SQL Server 2014+ (prior to that online rebuilds for partitioned indexes is only available for all partitions)

🔥 INFO: Enterprise Edition required for ONLINE rebuilds. REORGANIZE is an online operation and doesn't require Enterprise Edition, or the ONLINE keyword, however.

ALTER INDEX (Columnstore Indexes): Online By Another Name

Usage: maintenance operations (some details in the text below)

Documentation
“Starting with SQL Server 2016 (13.x), rebuilding a columnstore index is no longer necessary in most situations since REORGANIZE physically removes deleted rows and merges rowgroups. The COMPRESS_ALL_ROW_GROUPS option forces all OPEN or CLOSED delta rowgroups into the columnstore which previously could only be done with a rebuild. REORGANIZE is online and occurs in the background so queries can continue as the operation happens.”

In other words, for most purposes for columnstore in 2016+, use REORGANIZE when you want REBUILD ONLINE 🙃

ALTER TABLE SWITCH PARTITION: Online Esque

Usage: loading new data or removing data in “sliding window” scenarios

WAIT_AT_LOW_PRIORITY in Azure SQL DB, SQL Server 2014+
Documentation (search for “switch”)

🔥 INFO: This technically doesn't have the ONLINE keyword, but the WAIT_AT_LOW_PRIORITY option makes it online-ish (or at least less offline, if that's a thing).

What Exactly Would You Say You Do Around Here?

help@KendraLittle.com (Kendra Little) — Mon, 09 Dec 2019 00:00:00 +0000

The other day, I was looking back at an excellent blog post my colleague Jamie Wallis wrote about what Product Marketing Managers do at Redgate. I really like the chart he created which explains how Product Marketing Managers work with Product Managers – what each role focuses on, and where they collaborate.

I realized that my own role as an Advocate can also be hard to understand.

I did a podcast episode a while back on the topic – at the time, my role was called “Evangelism” rather than “Advocacy” – but I don’t expect my colleagues to listen to a podcast to figure out what value I can provide.

Why Does It Matter If People Know What You Do?

I work in a very collaborative role, but I’m not special: all roles which work with databases are increasingly collaborative. Even if you act more as a service provider than a partner for a group, it is very beneficial for you to help important groups in your business understand the value you provide to the organization. Taking the time to do this helps build your relationships with others:

They have a way to understand your team’s mission and strategy (rather than guessing based on assumptions or scattered interactions)
They can see how to best make use of your time
They have a better understanding of the value you provide to the larger organization – and that will benefit you whenever they talk about your group to others in the company. They will automatically help you connect with others and provide more value.

There’s another hidden benefit of writing this out. It may remind you that your job is a little bit cooler and more important than you realize, by helping you see the big picture.

Here’s an Example for My Role at Redgate

I asked Jamie if I could borrow and the diagram which he used in his post. He was happy for me to use it. One of the key partnerships that I have at work is with Redgate’s sales team, so I thought specifically of how I work with them when creating this first draft at outlining what I do, and how I can help them.

In this case, the sales team is my audience for explaining how I work, so I didn’t need to outline what they do as much as I needed to outline what I do on my own, and in collaboration with their group.

There’s a few way to share things like this which can help:

If you work in a physical office, print it and post it!
Share with new colleagues when they join your organization to help them get oriented
Add to slide decks in regular meetings to help introduce your team (even if you don’t spend time on the slide life, it can be a useful reference in shared PDFs)

By the Way, I’m Not a Team Lead

I’m not “in charge” of much of anything but myself these days – and I’m totally OK with that. (Managing other people is hard work!) I still went to the trouble to make this diagram. I’ve shared this with my team for input. I also shared this with my manager and let her know how I’d like to use this internally with our sales teams to help them understand our role (especially new folks coming on board, but also people whom I just haven’t worked with as much yet).

Earlier in my career, I tended to see activities like this as something that a team lead or manager would do. I didn’t really see it as my business.

Now, I have changed, and the world has changed a bit, too. It’s really everyone’s business to identify opportunities like this to represent your work a little better. Whenever you find a chance to do something to help connect to your colleagues and customers more clearly, it’s worth mapping out a little time to take care of that task yourself!

Installing Redgate SQL Toolbelt with Chocolatey (Video with Transcript)

help@KendraLittle.com (Kendra Little) — Mon, 02 Dec 2019 00:00:00 +0000

Chocolatey is a package manager that helps you install, upgrade, and uninstall packages (applications) on Windows quickly and easily from the command line.

Spoilers:

The command I show in the video is simply: choco install sqltoolbelt
Fun fact, you can also upgrade: choco upgrade sqltoolbelt
And uninstall: choco uninstall sqltoolbelt

The rest of the 6.5 minutes is about related tools you may want to install, and why this is all needed.

Links from the Video: Check These Out While You Watch

Gists with sample commands for useful installs for Microsoft Data Platform database folks:

Aaron Nelson’s List
Kendra’s List (forked from Aaron’s to track my own preferences)
Rob Sewell’s List

Video Explaining Chocolatey and Showing How to Install SQL Toolbelt (6.5 Minutes)

Transcript

Hello and welcome to this live stream episode.

My name is Kendra Little and I am a DevOps Advocate at Redgate.

Today I’m going to show you something that is a little bit life-changing for me. I have been setting up a new desktop and I’ve got a lot of apps to install, including things like Redgate SQL Toolbelt.

I remembered that I heard recently from my school my teammate Grant Fritchey, who’d heard from Steve Jones (also on our team) that when you’re setting up Windows the Chocolatey package manager can save you a bunch of time. I’d used Chocolatey briefly for setting up some tooling given out by our dev team for some cool preview stuff we’re doing, but I hadn’t really thought about the fact that it could save me a ton of time setting up Windows desktop apps. And it totally can.

So I want to show you a little bit about using Chocolatey to set up SQL Toolbelt.

So far today I have– just using the command line– installed SQL Server Management Studio, Azure Data Studio, Azure Data Studio PowerShell. All of the stuff you see on here, I have just installed using the command line.

I’ve even be able been able to customize installation. For example, I like to put the Sysinternals tools – these are tools like ZoomIt, Process Explorer, and Process Manager from the Sysinternals team at Microsoft— I like to put these in a custom folder. It’s where I always put it. I can actually pass in a parameter for that!

Sample Commands for Chocolatey Installs for SQL Server DBAs: Some Lists!

So these are a lot of the commands that I like to use: https://gist.github.com/LitKnd/93d02119cb10cef6992fd1bcddbdc73a

I forked this off a really useful gist from Aaron Nelson who has his favorite things to install with Chocolatey, too.

You can check out the things that Aaron likes to install, you can check out the things that I like to install.

All I did was I create a fork from Aaron so I could customize mine.

How to Install Redgate SQL Toolbelt with Chocolatey

I just uninstalled SQL Toolbelt (before we started). Chocolatey is a package manager and it’ll help you both install packages and uninstall them.

Now I’m going to change my command to: choco install sqltoolbelt

This saves you so much time installing packages– because look it’s going out, it’s finding the latest version of the Installer, grabbing it for you, and then starting to run it for you.

Part of the magic here is that you have your list that you can maintain in something like a gist or source control, so you don’t even have to remember what all of the things are that you want to install every time. Then you can simply paste them in.

Then also you don’t have to step through all of the wizards. You don’t have to download them to directory, click Next Next Next Next finish. You will notice that for some things, like SQL Toolbelt it will pop up a window (an unattended install window) for you— it’ll let you know how it’s going. It really depends on how the Installer is written as to how that happens.

Certain things may take a little bit of time– like everything else it can take some time– but this really takes all the work out of it. You can just get it going, start working through your email for the day, or maybe figure out how you’re going to configure your livestream on your new computer and just have everything set up unattended in the background.

I’m really excited for this brave new world. I never thought I would see the day where with Windows it was this easy. We can see over here that SQL Toolbelt has been installed successfully. Freya and I can now move on and get going with our day.

My Git CLI Cheat Sheet

help@KendraLittle.com (Kendra Little) — Wed, 27 Nov 2019 00:00:00 +0000

I created a cheat sheet for the Git Command Line Interface to go along with my Git tutorial for SQL Change Automation video.

I find the Git CLI to be very friendly and easier to learn than a GUI interface.

Setup

Download git CLI: https://git-scm.com/downloads

git clone <url>

Clone a remote repo (maybe an empty one) into your local directory. Ref.

git init

If not cloning, you might initialize a fresh repo locally. Ref.

.gitignore file

I often show people how to use Git with Redgate’s SQL Change Automation. For that tool, you can download and unzip the .gitignore file to the folder where your .sqlproj file lives: https://documentation.red-gate.com/display/sca4/Version+control.
Another great resource for figuring out what to ignore is http://gitignore.io/

Essentials

git status

Lists modified or added files, with status as to whether they are staged. Ref.

git add .

Stages all modified or added files. You must stage before you commit. Ref.

If you want to unstage everything: git reset Ref.

git commit -m “this is my meaningful message”

Commits staged files. Ref.

git push

Pushes staged files to remote repo. If upstream branch doesn’t exist, git CLI prints syntax to push to upstream branch with same name as your local branch: simply copy and paste! Ref.

git pull

Fetches from remote and updates current branch. If using feature branches, remember to checkout master and pull after you merge a Pull Request on the upstream repo. Ref.

Branching

git checkout -b features/mynewfeaturename

Switches to a newly created branch (the -b tells it to create a new branch. In Azure DevOps (and some other systems), the “/” means that this branch will be logically grouped into a features “folder-like-thing” and the branch will be named mynewfeaturename. Ref.

git merge master

Merges the contents of another branch (in this case, it’s master)into your current branch. Ref.

git checkout master

Switches to existing branch named master (trunk). Ref.

git branch

Lists all local branches. Ref.

git branch -d features/mynewfeaturename

Deletes local branch features/mynewfeaturename. Only works if you are not in that branch. Ref.

Stashing

git stash -u

Put away / temporarily removes all modified files. The -u means it applies also to even recently added untracked files (not yet staged). Ref.

git stash pop

Unpack your stash. Doesn’t matter if you’re in a different branch from where you stashed. Ref.

Upcoming Free Webinars on Database Development and Operations

help@KendraLittle.com (Kendra Little) — Tue, 19 Nov 2019 00:00:00 +0000

Fall is in swing, and it’s officially webinar season! Here’s a bunch of free events I’ve got on my calendar.

Weekly Database DevOps Live Chats: A New Experiment on YouTube

Stop by to chat! In this session I’ll be talking about what I’ve learned lately, what’s new in Redgate products, and whatever comes up with those who stop by live.

Weds, Nov 20th - 9:30 am Pacific - YouTube Link (30 minutes)
Weds, Nov 27th - 9:30 am Pacific - YouTube Link (30 minutes)

How to Make Your 2020 Monitoring Strategy a Success

Tomorrow, tomorrow, tomorrow! I’ll be chatting with BMW’s Tony Maddonna about monitoring in the real world. (The invite says my name is Arneh Eskandari, but I’ll be subbing in for Arneh as he’s been called away to Jury Duty.)

Wed, Nov 20th - 8 AM Pacific / 11 AM Eastern - Register

Essential Practices for High Performing Database DevOps Teams

Learn the four essential practices and see demos of these practices in action in this free session.

Tue, Nov 26th - 8 AM Pacific / 11 AM Eastern - Register

Why the Database Is at the Heart of DevOps Success

Join me and William Durkin from Data Masterminds for insights about how to succeed at database DevOps.

Fri, Nov 29 - 6:00 AM Pacific / 9:00 AM Eastern / 3PM CET - ScaleUp 360 online conference - Register

Managing and Automating Test Datasets for DevOps

I’ll be talking about data with Jeffrey Palermo of Clearmeasure. Join us!

Weds, Dec 4 - 7:30 AM Pacific / 10:30 AM Eastern - Register

Links and Resources from How to Architect Successful Database Changes

help@KendraLittle.com (Kendra Little) — Wed, 06 Nov 2019 00:00:00 +0000

Steve Jones and I had a great time today talking about source control for databases and release patterns for performance and availability in Seattle. We had a group of folks who asked terrific questions, made thoughtful comments, and interacted throughout the day.

Here are some links and resources from the day…

Suggestion for Configuration Option to Make SQL Server Developer Edition Act Like Standard Edition

This has been requested by the community for many years, but now is the right time to please vote up this suggestion.

After all, Microsoft just made Transparent Data Encryption as well as a whole slew of other cool things available in Standard Edition in SQL Server 2019. Things have really changed in Microsoft, and your voice matters: voting and commenting helps!

🔥 INFO: Standard Developer Edition has been added to SQL Server 2025.

The Unnecessary Evil of the Shared Development Database

Troy Hunt’s amazing post from 2011 is unfortunately still as relevant today as ever. Lots of helpful info on the business value of dedicated development environments is here: https://www.troyhunt.com/unnecessary-evil-of-shared-development/

Lock Escalation Magic Numbers

5,000 is a magic number… but it’s not necessarily 5,000 rows we are talking about, because more locks count than you might think.

How to Set Up a Self-Hosted Build Agent for Azure DevOps

This has come up twice in the last two days, so I think a video will be in the works for this! In the meantime:

An overview of Azure Pipelines agents: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents
If you wish to build and / or deploy to a SQL Server on-prem or build with something other than LocalDB, a self-hosted Windows Agent is likely what you want to set up. It works just fine to set that up on your laptop or a VM if you simply need to learn and test and it’s fine for it to be offline when your laptop or VM is shut down: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-windows
- I haven’t tested this new Learning Module on the topic, but it looks interesting. Note: the exercise is specific to using a VM in Azure to host the agent, not on-prem.

The “Aqua” Deployment Model

This series of posts on online deployments from Michael J Swart is soooooo good (and the cartoons can’t be beat, either): https://michaeljswart.com/2018/01/100-percent-online-deployments-blue-green-deployment/

My Favorite Whitepaper and a Helpful Update: Minimal Logging FTW!

Not a ton of people in the room knew about the amaaaaazing whitepaper on minimal logging in SQL Server: The Data Loading Performance Guide. This is important when you need to bulk load data fast.

Some guidance changes for SQL Server 2016+, for info on that, check out this post: https://blogs.msdn.microsoft.com/sql_server_team/sql-server-2016-minimal-logging-and-impact-of-the-batchsize-in-bulk-load-operations/

Accelerate State of DevOps Reports (DORA / Google)

Only a few folks in the rooms (whom I referred to as my fellow DevOps nerds with the UTMOST respect) knew about these free online reports which represent the findings from more than 30,000 respondents! This is the work led by Dr. Nicole Forsgren, and it is amazing. The DevOps Research and Assessment Group is now part of Google, download their work here: https://cloud.google.com/devops/

DBATools.io

Some PowerShell steps in our demos featured cmdlets from this wonderful and amazing open source community resource.

Azure Data Studio

Our demo has a narrative outlined in a Jupyter notebook from Azure Data Studio. Please don’t blame me for not using a PowerShell notebook, those were just released!

Resources to Download from the Day

If you are in need of the resources from the day, download them here (73MB: we use a lot of images and they only compress so much).

If you weren’t at the conference, feel free to download a copy. Please remember, though, that slides contain shorthand to help the speaker deliver a narrative, not full explanations: it’s easy to misunderstand based on slides without having been there. In other words, your mileage will vary, but perhaps you can find inspiration.

Thanks everyone for attending!

Why I Make Animated Gifs for Presentation Demo Backups

help@KendraLittle.com (Kendra Little) — Wed, 30 Oct 2019 00:00:00 +0000

PASS Summit 2019 starts up next Monday with pre-conference sessions. I’ve got my schedule all set, and I’m going to be busy: I’m speaking in two full-day pre-conference sessions, giving two regular sessions, judging the ever-exciting Speaker Idol competition, and also spending time at the Redgate booth in the expo hall. I scored a little PASS-TV interview to talk about Redgate right before the keynote on the first day.

I am super lucky to have all these awesome activities lined up, but one thing is clear…

I Need to Be Prepared

I will not have a lot of time to react if something goes wrong, such as laptop failure, a critical VM that won’t boot up, poor internet connection, etc. When it comes to my talks, I keep copies of my presentations in the cloud and on USB keys in case my laptop fails: that’s no big deal, as the conference provides workstations that speakers can use which are loaded with PowerPoint. However, demos are a different story.

My Old Way of Managing This Was to Create Screenshots of My Demos

I would then add the screenshots to hidden slides in my presentation. If I couldn’t do the demos live or if something weird went wrong, I’d un-hide the slides and start showing screenshots of how it was supposed to work.

The problem: screenshotting is time-consuming. And doing a run-through where you create screenshots breaks up your flow: the constant pausing, screenshotting, and pasting is very distracting. It’s not great demo practice.

My New Solution Is to Record a Run-Through of the Demo, Then Save It as One or More Animated GIFs

This means that I don’t need to pause, I can simply rehearse the demo while recording my screen: and I can think about important things, like what are the most useful points to convey about the demo to my specific audience at this event. There is no need to capture sound: if I use the gifs, I’ll narrate them live and talk through what is auto-playing on screen. You can even pause a gif that is playing in PowerPoint by clicking the menu at the bottom left of the screen in a live presentation.

Here is the workflow I use:

I use Camtasia to record my screen while I run through the demo. I have been a Camtasia for Mac user for years and find it very easy to use.
I save the project, make sure I have the resolution set to 1080p and the demo is full-screened, and do any simple edits needed (such as small cuts of where I made a mistake that’s not relevant)
I add a fade transition to the beginning and end of the clip to make it clear where it starts and ends (the animated gif will repeat when open on a slide in PowerPoint)
I publish the file as a gif (I use frame rate 5) and save it in the file share with my presentation, where it automatically syncs to cloud storage
I also add the gif file into the presentation itself on a hidden slide

For pretty short demos (less than a minute) I have even found that it’s convenient to show the animated gif and talk through it instead of doing a live demo. This is nice because it reduces switching in and out of PowerPoint when I want to show something quickly: screen switching takes time and can be visually jarring for the audience on many projectors.

One Downside of Using Animated GIFs This Way

Your PowerPoint files can become quite large. This was also a downside of my screenshot approach, so I’m already used to large PowerPoints. If it is an issue for you, you could simply not add the animated gifs to the PowerPoint and store them and access them separately if needed.

All in all, I’m quite happy with my updated method of “accident proofing” my talks. Long live the animated gif!

Thinking About Robert Davis

help@KendraLittle.com (Kendra Little) — Fri, 25 Oct 2019 00:00:00 +0000

It’s been more than a year and a half since Robert Davis passed away. I wrote a bit about Robert’s passing last April.

I haven’t written about him since, but I think about Robert a lot.

When I Remember Robert

I think about Robert when I work on a tough problem that he would have found interesting, when chatting about database nerdery on Twitter, when preparing for a conference or training that I know he would have been excited to attend.

I remember him when I get a great new photo of one of my dogs, because I know he would have loved to see it.

When I remember Robert now, I remember how delighted he was to solve all sorts of puzzles, and I remember the sound of his laughter. He helps me try to remember to be kind. To be more helpful to others.

A Simple Remembrance

I don’t have a big takeaway or piece of insight from this post. I simply wanted to write something about our friend who isn’t with us anymore, because he continues to make an impact on my world.

Wish you were still here.

Building My Schedule for PASS Summit 2019

help@KendraLittle.com (Kendra Little) — Thu, 24 Oct 2019 00:00:00 +0000

It’s just ten days until PASS Summit 2019 begins in Seattle. The schedule is up and there are loads of good sessions. Here’s what I’m putting on my calendar to make sure that I don’t miss it: along with some sessions I wish I could attend that I’ll be sure to catch the videos of afterwards.

Monday, Nov 4: “Redgate SQL in the City Summit Seattle” Pre-Con Day - Room 3AB

This is going to be a fantastic day: loads of DevOps learning and sharing. I’ll be talking about implementing and improving Database DevOps with some of my very favorite DevOps and SQL Server specialists: Steve Jones, Grant Fritchey, Ike Ellis, Hamish Watson, and Arneh Eskandari. The great thing about having a list of speakers like this is that you get to hear about varied real-world experiences and different takes on topics, plus there is loads of time to bounce ideas off people. There’s still time to sign up and join us!

Tuesday, Nov 5: “How to Architect Successful Database Changes: Source Control and Release Patterns for Performance and Availability” Pre-Con Day - Room 604

On Tuesday, Steve Jones and I will be talking about development patterns that enable you to frequently release database changes while maintaining high production uptime and minimizing the impact of your deployments on your target environment. This isn’t the easiest thing to do, but it’s increasingly important to understand the methods behind sneaking in database deployments to live systems without causing problems for users: we’ll share what we’ve learned over the course of our careers, as well as some insights into what we see as the future of database development.

Tuesday evening: There is a Women in Tech Happy Hour from 4-6 PM at the Tap House Grill. I’ll be late, but it’s a great event I’d like to make, even if it’s just for 15 minutes. Following this, there is a Welcome Reception in the Keynote room back at the Conference Center. I will almost certainly head back to do a little preparation for my presentation the next day and an early night of sleep, however: after speaking a lot during the day I’ll be tired and my voice and brain will need a rest. (Note: sometimes people feel bad for taking nights off networking at conferences: fear of missing out can kick in pretty easy. But if you need time to chill and recharge, do it!)

Wednesday, Nov 6: Day 1 of the main conference

08:15 AM - 09:45 AM: Keynote with Rohan Kumar. I’ll watch this remotely on PASS TV, as I am presenting a session directly afterward. If you can make it in person, everyone will be walking to the big Keynote Room, follow the crowd.

10:15 AM - 11:30 AM: I’m presenting “Why the Database is at the Heart of DevOps Success” in room 615 (it was 608, but we had a room change). I love, love, love this session and am so excited to share it.

11:30 AM - 01:30 PM: I’m going to swing by one of the DevOps tables at the Birds of a Feather lunch (Dining Hall, 4EF) for a quick chat, then head to the Redgate booth in the Exhibit Hall (4B).

1:30 PM - 2:45 PM: I cannot possibly miss Pam LaHoud on “Improving Availability in SQL Server and Azure SQL Database with Accelerated Database Recovery and Resumable Operations.” She is magic and these new features are magic, too. Room 6B.

2:50 PM - 3:30 PM: Back to the Exhibit Hall (4B): I love hearing questions and chatting DevOps at the Redgate booth.

3:15 PM - 4:30 PM: The Exhibit Hall closes at 3:30, so I’ll probably be late for this. I’ll try to find a spare seat in Linux Fundamentals for SQL Server DBAs by Hamish Watson in Room 608.

04:45 PM - 6:00 PM: It’s SPEAKER IDOL time! Round 1. This is a really fun series of lightning talks with a fun competition mixed in. I get to be a judge! I can’t wait, I learn a ton from Speaker Idol every year. Be there in room 2AB.

6:00 PM - 8:00 PM: Have some food and drinks on the Exhibitors: and maybe talk to us as well if you like, we’re nice enough - Exhibit Hall, 4B

If my voice isn’t already starting to fail, I’ll head to Game Night at 8PM (Ballroom 6A).

Thursday, Nov 7: Day 2 of the main conference

08:15 AM - 09:45 AM: Keynote with Tarah Wheeler (Keynote Room). Imma try to attend this one in person. I need to make notes for a post-event webinar that I’m doing with Grant, Kathi, and Steve: “What we learned at PASS Summit.”

10:15 AM - 11:30 AM: “Enabling End to End Scenarios with Data Virtualization in SQL Server 2019” with James Rowland-Jones, TCC Yakima 1. It’s a hike over to the TCC building (you can do it all inside the convention center using a skybridge for convenience), but this is something I must learn more about for cross-platform scenarios. And isn’t this a crazy world: James works at Microsoft and he’s presenting on how to use ORACLE AND SQL SERVER TOGETHER! Oh brave new world!

11:30 AM - 01:15 PM: Women in Tech Lunch with LaShana Lewis (Keynote Room). I can’t remember if I signed up for this when I registered, but I’ll find out when I check into the conference. If I didn’t, I’ll attend the Birds of a Feather lunch (Dining Hall, 4EF) and catch LaShana’s talk via recording afterward. Hopefully will fit in some time at the Redgate Booth in the Exhibit Hall.

1:30 PM - 02:45 PM: SQL Server in Containers: Your Next Dev/Test/Integration/Upgrade Server. Grant Fritchey has some amazing things to show and I’m going to tweet all about it. Room 618.

2:50 PM - 3:30 PM: Back to the Exhibit Hall (4B) to meet up with friends at the Redgate booth. Exhibit hall closes at 3:30 again.

3:15 PM - 04:30 PM: I’d love to see Tobias Ternstrom talk about Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases. Room 3AB.

4:45 PM - 06:00 PM: SPEAKER IDOL - ROUND 2 in Room 2AB. The action heats up in this second round with an all-new set of contestants. A great community session.

I have an early session Friday, so I’ll be heading to dinner with Friends of Redgate and going to bed early. I wish I could go to Game Night at 8PM (Ballroom 6A) as well, though.

Friday, Nov 8: Final day of the conference

8:00 AM - 09:15 AM: I’ve gotta see Batch Execution Mode on Rowstore Indexes by Niko Neugebauer in room 6B. I will have to watch the recording of Niko’s excellent talk because SPEAKER IDOL - ROUND 3 is in Room 2AB. Start your day off with a fun sampler of learning in this final heat of the first round.

9:30 AM - 10:45 AM: I’m presenting “Best Practices for Branching Database Code in Git” in room 6A. WOO HOO!

10:30 AM - 2:00 PM: Exhibit Hall (4B) - I can’t show up at 10:30 unless literally no-body is in the audience of my talk, but I’ll get there as soon as I can for the last day at the booth.

2:00 PM - 03:15 PM: SPEAKER IDOL - FINALE, Room 2AB. A winner will be crowned! Errr… actually I don’t think there’s a crown, but there IS a very cool prize.

3:30 PM - 04:45 PM: I wish I could see Continuous Intelligence… What’s This All About? by Gabi Münster in TCC Skagit 4. Unfortunately I have to jet to the airport to get home to my puppy, who I’ll be missing like crazy all week. I know, excuses excuses.

Workshop Report: Overcoming Obstacles on the Journey to Continuous Delivery for Databases (Video with Transcript)

help@KendraLittle.com (Kendra Little) — Thu, 17 Oct 2019 00:00:00 +0000

In this video, Freyja the puppy and I talk about a recent workshop which I facilitated at the IDC DevOps conference in London.

We cover:

01:45 The workshop methodology: about the Lightning Decision Jam Session, by AJ&Smart / Jonathan Courtney. I think this is a great format, and you may want to use this yourself for a workshop with your team (on any problem-solving topic)
05:33 Key findings from our group about how to overcome the top obstacles to Continuous delivery for databases

A transcript of the session is below the video.

Transcript (edited a bit for clarity)

Hi, I’m Kendra Little: I’m a DevOps Advocate for Redgate

What this means is that I get to spend time talking to folks and thinking about the patterns and practices that help people successfully develop and deploy stable changes to their databases, that bring value to the customers of these organizations.

Recently I went to a conference that was really great: it was the IDC DevOps conference in London. At this conference, I really liked the format. In the morning we had a series of presentations, and in the afternoon we had workshops where we talked about different problems and potential solutions.

These workshops were collaborative

I hosted a workshop, but I didn’t get to just get to tell the whole group: here’s how it’s done. That wasn’t the point!

The problem that we were solving in my group was how to overcome the top obstacles to continuous delivery for databases.

What we zoomed in on was the fact that when you’re starting out on a journey to implement Database DevOps, you’re working across silos. You’ve got the database folks generally working in their own silo. You’ve got developers generally working in another silo, and the larger the organization the more complex it can be to establish new processes between these two groups, because they have different ways of working.

Starting out at the workshop, a lot of people at the conference were in different phases of implementing DevOps, and many were early on. So we did this exercise that not everybody was bought into at first.

This exercise is built on a format called the Lightning Decision Jam Session by AJ&Smart, and that itself is based on a great book called Sprint. It’s about design thinking.

Here’s a Little Bit About How the Workshop Goes

You have everyone take a stack of either paper or post-it notes: having something sticky works well though. Everybody privately writes down different potential solutions to how could we make it easy to get people to take on the new processes required for database DevOps.

1. Generating Solutions

Everyone generates these solutions. You focus on quantity, not quality. The idea is that everyone should just start generating ideas, because it’s like brainstorming but without the pressure of having other people listen to your ideas. You’re just recording them on sticky notes. You have a time limit for that, and we did about ten minutes of people generating their solutions.

At first some people were like, well, I don’t know what solutions I have for this because I haven’t done Database DevOps successfully!

But the truth is most people in the group had gone through IT changes and seen what worked well and what didn’t. Most people in the group had observed a lot of behavior among their co-workers and seen what’s difficult and what’s not difficult to get people to do. And also a lot of people at the table had gone through Agile transformations for their application tier, and had seen what worked and didn’t work for that. Everyone already had a lot of reference material to work from.

2. Voting

After you generate all these solutions, you come together. Each solution needs to be legible because you’re not gonna defend them, you’re not gonna explain them. What happens is the facilitator puts them up on a board: we just used a wall in the room. You spread them out on the wall and you equip everyone with either voting dots or a pen. Everyone reads silently reads all the solutions and puts dots on the one that any of them that they think would be viable.

Now, you’re not limited. You can vote for as many of these as you want, but the point is that you don’t vote for your own because you already submitted yours: you already kind of voted for your own. Everybody silently votes on these using dots. The idea is not to do a bunch of talk during this, just read them and vote on them.

3. Filtering

Then you do a filtering process. We filtered out anything that had less than three dots. These aren’t terrible ideas, necessarily, these are just things that the group doesn’t think are gonna be the most effective. That narrows down the window of what you have.

4. Effort / Impact Scale

Then you take down the ideas that pass the dot filter, and you arrange them by the amount of impact they have in the amount of difficulty they’ll be.

This is where having an axis drawn on a whiteboard can come in handy. We just had to imagine the axis ourselves, but ideally you’ve got a vertical access which is the amount of impact a suggestion will have on the organization / the amount it’ll help you solve the problem. The horizontal axis is how difficult it is.

Things that are very far to the left and very high up will be very high impact and not a lot of work. You can also have things that are high impact and a lot of work (top right quadrant). You can have things that are low impact and a lot of work (lower right quadrant). You have things that are low impact, but they are not a lot of work, either (lower left quardant).

We arranged all the ideas collaboratively on this scale.

This led us to identify things that are high impact but not super duper challenging to do.

The Findings That the Group Came Up With Were

1. Version Control

For the top three things that help people overcome process obstacles to initiating change for continuous delivery for databases, our group found that establishing version control for database code is number one.

Version control is the foundation of so much collaboration between people that you really need to get code into version control and focus on that. This isn’t always the simplest thing in the world, but it’s really a necessarily necessary enabler for everything else.

2. Minimum Viable Dataset

The group also had a concept of a Minimum Viable dataset. I loved this point: in order to successfully do development work, we need a copy of a dataset that enables progress.

In some cases, if production is a 10 terabyte environment our Minimum Viable Dataset for development might be much smaller. This viable data set may need to replace sensitive data that lives in production with non sensitive data. In some environments it may need to be synthesized. But we need to have this to support this work.

3. Constant Communications Between Teams

The third thing that the group found is this enabler to break down these silos between teams and get database DevOps going is constant communications between the teams. We need to have teams really working together to establish these new processes, and we need to have the teams listening to one another.

Additionally, in order to enable all of these things to happen, if you can identify an executive sponsor who can meet with the teams on a regular basis and emphasize how this project meets organization goals, it helps these teams to work together.

For cultural change, it’s good for teams to know they are being supported by the organization at large. That is an underpinning that can help this change go through.

This was a really fun workshop at the IDC DevOps conference. Every time I’ve done this exercise, I’ve been impressed at the insights that come from the group: which is always so much fun because at the beginning the exercise there’s so much doubt as to whether the team knows enough to overcome the challenge! But in this case, we certainly did.

Thanks for joining me for this talk. I’m Kendra Little from Redgate: and this is Freyja, our local expert on Corginetes who has joined me for this quick video. Bye folks, I’ll see you again in another video soon.

Setting a Custom Variable in an Azure DevOps Pipeline with PowerShell

help@KendraLittle.com (Kendra Little) — Mon, 02 Sep 2019 00:00:00 +0000

Recently, I was doing a bit of work in Azure DevOps Services, preparing a demo for an upcoming webinar. I ran into a simple but frustrating problem.

Part of the demo does the following magic, using a branch policy and pull request automation trigger, combined with some of Redgate’s extensions:

Builds/validates database code
Creates a lightweight clone of the “production” database (I’m using a copy of StackOverflow, thanks Brent & the folks at Stack)
Creates a release artifact summarizing the changes that’ll be deployed to the clone, exports it, then deploys the changes

This combination of actions is lovely: reviewers of the pull request have validation that the code builds, they know it deploys successfully, and they can even look at deployment timings. Plus, they have a real environment to review the change.

I was working on improving something simple about my demo: giving the cloned database a clear, accurate name that ties to the Pull Request Id.

Previously, I’d been using the default $(Release.ReleaseId) variable in the database name, but I decided that I would prefer to identify this with the PR number, and for the cloned database to be replaced when re-running the automation for a PR (if code in it changes, etc).

There Wasn’t a Default Variable That Does Exactly What I Wanted

Looking at my options in the default variables, the closest thing to what I was going for is $(Release.{alias}.SourceBranch)". My alias for my build artifact is _StackOverflow-CI, so I referenced that as $(Release.Artifacts._StackOverflow-CI.SourceBranch).

That outputs a path like refs/pull/34/merge. For a variety of reasons, I don’t want a bunch of forward slashes running around in my database names. But no problem: looking at the documentation, I saw that I can set a custom variable easily in a PowerShell task step in my pipeline. Maybe there’s an even simpler way to do this, but that looked like a fast and easy way to replace those slashes with underscores, so I went for it.

A Key Piece of Info: New Variables Are Only Available in Downstream Steps

This ended up being a bit time consuming for me, because there’s one important piece of the documentation which I didn’t notice. I skipped to the sample code and missed this:

To set a variable from a script, you use the task.setvariable logging command. This does not update the environment variables, but it does make the new variable available to downstream steps within the same job.

Azure DevOps Services - Variables doc - emphasis mine

Having missed that fact, I struggled with my code for a good while, because I was trying to set the variable and then read it for validation in the same task. That didn’t work, so I thought I wasn’t actually setting the variable properly. Whoops.

Some Simple Sample Code

Let’s say you have a PowerShell script task in your Azure DevOps release pipeline. You set it to “inline” type, and you have the following code in there:

$path="$(Release.Artifacts._StackOverflow-CI.SourceBranch)"
Write-Host "$path"
$path= $path -replace "/", "_"
Write-Host "No problem reading $path"
Write-Host "##vso[task.setvariable variable=DUCKS]$path"
Write-Host "Trying to read $env:DUCKS"

This will produce something like this:

2019-08-14T20:56:19.8412596Z refs/pull/34/merge
2019-08-14T20:56:19.8424967Z No problem reading refs_pull_34_merge
2019-08-14T20:56:19.8431236Z ##[debug]Processed: ##vso[task.setvariable variable=DUCKS]refs_pull_34_merge
2019-08-14T20:56:19.8545546Z Trying to read

Totally looks like our $env.DUCKS variable didn’t get set.

However, if you add an additional PowerShell script task to the same job, also set to “inline type”, which contains that same final line:

Write-Host "Trying to read $env:DUCKS"

That will do what you expect and produce:

2019-08-14T20:56:21.4502122Z Trying to read refs_pull_34_merge

I Guess Reading Is Fundamental

This all makes perfect sense to me, now that I think about it.

In my defense, it’s been a loooong time since I’ve thought about environment variables in Windows! I couldn’t tell you how many times I looked at that doc and missed this important detail: it was more than a few times.

Hopefully this helps someone else out there who misses the same thing I did.

Git Command Line Tutorial with SQL Change Automation for SSMS (Video)

help@KendraLittle.com (Kendra Little) — Mon, 26 Aug 2019 00:00:00 +0000

I’m excited for Redgate’s new SQL Change Automation plugin for SQL Server Management Studio (SSMS).

SQL Change Automation lets DBAs and developers use a migrations-first approach to create precise scripts to apply changes to your database. If you’re curious about what I mean by “migrations-first”, read more about this approach, and how it compares to a state-first approach here.

I’ve been working with SQL Change Automation with Git for a while in Microsoft Visual Studio. Visual Studio contains a lot of integration by default with Git and Azure DevOps, so I’ve been using its graphical tools, for the most part.

The New SQL Change Automation Extension for SSMS Works with the Version Control System (VCS) of Your Choice, But This Initial Release Provides Only “Working Folder” Support

That means that you set up your version control outside of SSMS, then point SQL Change Automation at your working folder.

This inspired me to start learning the Git command line. I’m not a command line guru of any sort, but it turns out the Git command line is really easy to learn, and I think it’s super fun to use.

In this 22 minute video, I give an overview of how to create a fresh repo in Azure DevOps and clone it to your local machine, set up a new SQL Change Automation project in SSMS for the Microsoft pubs sample database, commit changes to source control, work with branches, and stash files away when you need to.

00:40 Where to get the code to create the Pubs database
01:00 Creating a new repo in an Azure DevOps project
01:39 Clone the repo to your local machine with: git clone urlfromyourrepogoeshere
03:00 git status
03:30 Create a new SQL Change Automation project and baseline it in SSMS
06:00 Stage the files with: git add .
06:40 Review staged files with: git status
07:33 Commit the files with: git commit -m “commit message goes here”
08:08 Send commit to upstream repo with: git push
09:42 Create a feature branch with: git checkout -b foldername/branchname
10:25 Add a new migration script with SQL Change Automation extension in SSMS
11:36 Observe how non-committed files in working directory act in Git when changing branches
13:04 Put away / temporarily remove “dirty” files with: git stash -u
16:02 List local branches with: git branch
16:11 Switch to an existing branch with: git checkout foldername/branchname
16:17 Unpack / get back your stash with: git stash pop
16:58 Stage modified files in our feature branch with: git add .
17:03 Commit modified files in our feature branch with: git commit -m “commit message goes here”
17:33 Get detailed syntax to push feature branch for the first time to remote repo with: git push
18:07 View repo in Azure DevOps Services
18:22 View branches and create a pull request to merge our feature branch into master
19:05 Complete pull request, note I use an option that deletes the feature branch on the central repo after merge
19:45 Validate that the feature branch still exists on my client machine with: git status
20:10 Return to master branch with: git checkout master
20:22 Clean up my local feature branch with: git branch -d foldername/branchname
20:40 Pull down updated code from master with: git pull

Key Findings for Database Professionals from the Accelerate State of DevOps Report 2019

help@KendraLittle.com (Kendra Little) — Thu, 22 Aug 2019 00:00:00 +0000

The Accelerate: State of DevOps Report 2019 has just been published. This report is the latest in six years of research. With more than 31,000 survey responses, Accelerate is the longest running study of DevOps in academia or industry.

In the 2019 edition, research continues to show that DevOps drives business value: high performers at DevOps are “twice as likely to meet or exceed their organizational goals.” While this isn’t a new finding, it’s very important that this continues to be true: why invest in improving at DevOps if it doesn’t drive business value?

While there are a ton of valuable insights in the report, in this post I will focus in on the findings which I believe are most relevant to those of us who work “close to a database.” There are three very interesting aspects of the research which hit close to home:

Speed and stability are not tradeoffs
Heavy change processes negatively impact speed and stability
Communities of practice are a common and successful tool to transform culture

Speed and Stability Enable One Another

Accelerate recommends tracking four key metrics, in addition to availability. Note that half of these are related to speed, and half are related to stability:

Speed		Stability
Lead time for changes	Deployment frequency	Change failure rate	Time to restore service

Accelerate explains that the relationship between these metrics is often misunderstood:

Many professionals approach these metrics as representing a set of trade-offs, believing that increasing throughput will negatively impact the reliability of the software delivery process and the availability of services. For six years in a row, however, our research has consistently shown that speed and stability are outcomes that enable each other.

Accelerate: State of DevOps 2019, p17

This is a tricky thing for IT professionals who have been in the field for a while to get our head around.

Another way to put this is that, as Microsoft found, “moving fast with poor quality is as bad as moving slow.” A raw deployment count doesn’t give any indication of delivery of value. If many of those deployments are doing rework and making up for poor quality, that needs to be factored in, because the point is delivering value to customers: hence the importance of stability metrics.

The important thing for IT leaders and data professionals to notice is that the evidence has surely piled up that DevOps techniques are a tool that helps us improve stability: and when it comes to data, that’s typically one of the biggest areas where a company cares a great deal about maintaining quality and stability.

There is a clear action for leadership to take here: moving to DevOps means building a team across development and IT silos, and making the metrics above universal outcomes which the entire team moves towards. You must stop rewarding IT based on stability, and stop rewarding software development based on feature delivery: all groups are working towards the full set of metrics.

Heavy Change Processes Negatively Impact Speed and Stability

IT and database specialists typically act as gatekeepers for production. Traditionally, when things go wrong, heads of IT and their related specialists try to make things better by adding another layer of process to things. This has resulted in complex change approval processes, often including a Change Approval Board.

As much as we like to think that a complex process will make things better by weeding out problematic deployments, Accelerate finds that in our attempt to make things better, we’re actually making it worse:

…we investigated whether a more formal approval process was associated with lower change fail rates and we found no evidence to support this hypothesis, consistent with earlier research. We also examined whether introducing more approvals results in a slower process and the release of larger batches less frequently, with an accompanying higher impact on the production system that is likely to be associated with higher levels of risk and thus higher change fail rates. Our hypothesis was supported in the data.

Accelerate: State of DevOps 2019, pages 50-51

This doesn’t mean that you must get rid of change management. Many organizations simply cannot do that due to regulations, and Accelerate also doesn’t suggest that as the path to success.

But we do need to work towards significantly changing the way that change management and change approvals are done. This includes clarifying change processes to make it incredibly clear to everyone what needs to be done to have a change approved. This also means transforming the jobs of specialists from production gatekeepers into consultants in the software development process: both to build the most effective change process, and to act as peers to review critical changes early in development, and shape the testing strategies for those changes.

Similarly, leadership on Change Approval Boards should shift into more strategic roles, identifying areas where capabilities are needed the most, and providing guidance and resources on how to improve.

Changes like this are not going to happen overnight, particularly in larger enterprises. Perhaps this is part of why Accelerate found this year for the first time that Enterprises, defined as organizations with more than 5,000 employees, comparatively are lower performers than smaller organizations. However, enterprises can change their culture and processes. On a personal note, I am frequently surprised at the massive level of transformation that I’ve observed from my friends who work at Microsoft over the last few years, for example. In an Enterprise situation, the game is about persistence and about seizing promising opportunities to work in new ways.

Communities of Practice Are a Common and Successful Tool to Transform Culture

One question I often get from database developers or DBAs is how they can convince developers to change their habits in one way or another. This is sometimes phrased as, “they don’t listen to me.”

Accelerate finds that if you want to change your culture, you’re probably going to need to get out of your silo. They asked respondents to share how DevOps and Agile is spread at their organizations, and that found that “Low performers tend to favor Training Centers (also known as DOJOS) and Centers of Excellence (CoE): strategies that create more silos and isolated expertise.” Accelerate: State of DevOps 2019, p71

We already have plenty of silos and isolated expertise when it comes to databases, we certainly don’t want to create more.

Instead, tools like Communities of Practice are found to be more effective. 57% of “Elite” performers were found to use Communities of Practice, the heaviest concentration of any single strategy studied.

Communities of Practice build both knowledge and community inside organizations. For IT leadership, it’s critical to encourage specialist groups in your organization to seed and sponsor these groups, and to participate in them as active community members. For specialists such as DBAs, establishing a Community of Practice around database development is your best tool toward transitioning from being a “production gatekeeper” into becoming an internal consultant who shapes how database development is done in your organization: a key contributor and collaborator in the software development lifecycle.

Want More Insights?

You can download the Accelerate: State of DevOps Report 2019 today.

Catch a recording of our webinar “What You Need to Know About the 2019 DORA Accelerate State of DevOps Report, with Jez Humble and Steve Jones”.

Azure DevOps: How to Set a Custom Release Branch Folder Filter in Build Pipeline Triggers (Video)

help@KendraLittle.com (Kendra Little) — Mon, 19 Aug 2019 00:00:00 +0000

This is the first in a series of posts about simple things that I had a hard time figuring out in Azure DevOps services.

It can be very useful to enable Continuous Integration for multiple folders in your DevOps pipeline: say, for every branch created under releases/ or features/. But configuring this can be strangely confusing!

The Solution

In this video, I show how to get a custom release branch folder set in your triggers for both build and release pipelines. Then I show a quick test of the automation at work in my demo pipeline.

Spoiler: the fix is to type directly into the filter box and hit enter. It looks like that box is only for filtering existing branches, but you can use it for edits! The fix starts in the video at 1:50.

SQL Server Management Studio Is as Relevant as Ever

help@KendraLittle.com (Kendra Little) — Wed, 14 Aug 2019 00:00:00 +0000

🔥 UPDATE: Azure Data Studio (ADS) was retired on February 6, 2025, with support ending on February 28, 2026. Additionally, SSMS has been modernized and is no longer based on the legacy Visual Studio shell—it's now built on Visual Studio 2022 with 64-bit support and improved performance.

For more details, see the Azure Data Studio retirement announcement and Erin Stellato's announcement about SSMS modernization..

Sometimes you keep a classic around.

After almost fifteen years of heavy usage by developers and database administrators (DBAs), it might seem like Microsoft’s free tool, SQL Server Management Studio (SSMS), is about to go out of style.

After all, SSMS is no longer the cool new kid on the block: Microsoft has shown consistent effort to develop their new tool, Azure Data Studio (the artist formerly known as SQL Operations Studio), since November 2017. Azure Data Studio is built on the modern foundation of Microsoft’s VS Code, whereas SQL Server Managed Studio is related to the legacy Visual Studio Shell.

Based on this overview, it might seem like a new SQL Server DBA or developer should primarily learn Azure Data Studio, not SSMS. And it might similarly seem like vendors should focus on developing new tooling only for Azure Data Studio.

But when you look into the details of how Azure Data Studio is being developed, it becomes clear that SSMS is still just as relevant than ever:

Azure Data Studio really shines where it specializes in different functionality than SSMS provides

While both Azure Data Studio and SSMS each provide an interface to author queries and to execute them against relational database instances, I find that the user experience in Azure Data Studio is often not nearly as smooth as it is in SSMS.

Where Azure Data Studio really shows its value is in unique experiences:

SQL Notebooks (based on Jupyter notebooks) which offer an experience of “interactive documentation” and more
The ability for users to connect, manage, and query different database platforms, using tools like the PostgreSQL extension
The ability for MacOS or Linux users to run Azure Data Studio natively, without installing a Windows client
The ability to work in other languages, such as PowerShell

Azure Data Studio is being built to sometimes connect with SSMS

One Azure Data Studio feature I noticed recently is “Release of Database Administration Tool Extensions for Windows” in June 2019. Here’s the overview of the feature:

This extension launches two of the most used experiences in SQL Server Management Studio from Azure Data Studio.

Release Notes for Azure Data Studio (emphasis mine)

As a MacOS user, I was slightly disappointed that this is a Windows-only feature: it’s Windows-only because it requires that SSMS be installed, so the SSMS feature can be launched inside Azure Data Studio.

However, I understand where this is coming from: I don’t need every SSMS feature to work natively in Azure Data Studio on MacOS, and I suspect that this saved a ton of engineering work. If Microsoft plans to support and develop both tools long-term (and I believe this is evidence that they do), it makes sense to strategically create these dependencies.

SSMS is still under active development

A major update to SQL Server Management Studio, SSMS 18.0, released in April 2019. This release included many improvements and new features. New features have been regularly added in the versions since this release as well.

This pattern shows evidence to support Dinakar Nethi’s suggestion in his SSMS 18.0 release announcement that we should “think of these two tools not as separate tools doing different things, but as one integrated tool. Each tool has different experiences built into it and can be launched from the other seamlessly.”

Users still care a ton about SSMS – and therefore, vendors care about it as well

One comment I have frequently heard over the years from developers and DBAs who work with multiple platforms is that they are impressed by the richness of tooling provided for the Microsoft Data Platform at no extra charge. SSMS provides a very rich experience and covers a vast amount of features – and we users very much enjoy it.

For this reason, vendors will continue to build new features for SSMS.

For example, at Redgate, we have just created a major new extension for SQL Change Automation in SSMS, which allows users to author changes in a migrations-first approach to development. We wish to empower teams to collaborate both across Visual Studio and other IDEs, and we recognize that SSMS continues to be the primary tool for Microsoft Data Platform DBAs and many developers – it remains incredibly valuable to offer users the ability to work in SSMS.

Where do we go from here?

SSMS remains the primary tool for SQL Server specialists. Azure Data Studio is a terrific, complementary tool, with strong use cases for cross-platform experiences and SQL Notebooks.

For new database administrators working with the Microsoft Data Platform, it continues to make sense to download and learn SSMS first – but do take a few minutes to install both tools and play around with the SQL Notebook functionality in Azure Data Studio specifically, these can be a great tool for you to document information as you learn. This is also true for developers who need to perform some operational tasks.

For occasional users who only need to connect to existing instances and run queries, either SSMS or Azure Data Studio is a fine choice.

For odd users like me, who generally run in a MacOS or Linux environment, and who periodically do more in-depth performance tuning and database management tasks, it’s especially valuable to become fluent in both tools. I love using Azure Data Studio for its Notebook experience and running occasional queries against SQL Server running in Docker on my laptop when appropriate, and only firing up a Virtual Machine to use SSMS when I need a more in-depth experience.

How to Persuade Your Company to Change

help@KendraLittle.com (Kendra Little) — Fri, 19 Jul 2019 00:00:00 +0000

Like a lot of developers and database administrators, I do a fair amount of short-term problem solving during the course of my normal work week.

I get to join some Redgate sales calls, often during Proof-of-Concept exercises, and this frequently involves helping brainstorm about the best way to use our tools to solve a specific need for monitoring, database development, or automation. I also pick up customer questions from the #Redgate channel in the SQL Server Community Slack and answer forum questions for Redgate when I can.

Inevitably, when you do a regular amount of troubleshooting and brainstorming with customers, you start to notice patterns and have ideas about how to make things easier, faster, or better in some ways. Sometimes this is an idea for a new process or a new product, or it might be a big change to existing processes or products.

The challenge is getting it to happen, especially when doing so requires time from your coworkers.

I’m getting close to my one year anniversary at Redgate, which caused me to pause and think critically about the biggest patterns I’ve seen with one of my favorite Redgate products: and about some changes I would like to see to make that product even more awesome. (I’m not going to tell you what the product or the idea is in this post, by the way. The point is how I’m trying to get the idea done.) I realized that this idea is pretty important to me, and I would really like to see it happen: but it’s not a simple or easy thing. So I started to think about what I could do to try to cause some change to happen in this area.

Saying, “Hey, We Should Do This Thing!” Usually Isn’t Enough

Occasionally, you can successfully initiate change simply by dropping the right idea into the right brains at the right time. But this is pretty rare, in my experience. Usually that “right time” described is when you can clearly and quickly show that the change will either result in a large amount of revenue immediately, or avoid a disaster which is provably imminent.

For anything else, even amazing ideas risk falling by the wayside. There are a lot of great ideas out there, but typically all our coworkers and managers already have a direction set for the next few weeks or months, if not longer.

It takes more than mentioning an idea to get it prioritized, much less actually done. So we need to put in more work to give our great idea a better chance at happening.

Think About the Way Your Company Likes to Communicate

Company cultures vary quite a bit. If you work at a startup, you’re going to pitch ideas quite differently than if you work for a government agency.

At Redgate, I’ve learned that people love writing and reading. You can see this about us externally by browsing Simple Talk, our Product Learning articles, and even our product documentation. We also internally use video quite a bit – we share video for live meetings and demos, and often share recorded meetings for distributed teams and learning afterward.

For my pitch, I decided to focus mostly on writing, but to use a bit of recorded video as well. I wrote a paper about the change I was advocating, and why it should be done.

Here’s what I included in the proposal.

Give an Executive Summary

My proposal ended up at around 11 pages, including the table of contents. This is fine for my audience: like I said, Redgaters like writing and research. Even so, not everyone has time to read 11 pages. Even if they do have time, you should start with a pitch about why they should spend the time with your proposal.

That’s what an executive summary does. Give a short, high-level, one-paragraph overview of what the problem you’re solving is, the impact change will have on your organization, and a suggested course of action.

Explain the Problem, and Talk in Dollar Signs Whenever You Can

The next thing I did was detail the problem I’m solving: what thing or things could be much better? How does my idea relate to those problems?

In my experience, the more you can quantify the opportunity you see in revenue, and the more specific you can be about that, the more of a response you will get.

In my case, I explained this in three steps:

The current experience, and why/where it could be improved
A list of customers who asked for something related to my idea, or who could / would have benefitted from what I am proposing
Details on how competing tools measure up in this area

This wasn’t something I did in five minutes: it took several research sessions. I dug through my email and identified some customers that way, and also asked colleagues for examples they remembered. I searched through Salesforce to identify more examples, and also to be able to quantify the size of deals which might be accelerated or created if my idea was implemented. I also did research on competing tools.

Be Open to Other Solutions

Possibly the hardest part about this whole thing is to not get married to your idea.

Yes, you’re proposing a specific thing. But the really important bit is actually bringing focus to the problem or opportunity that “thing” addresses. There may be ways this can be approached more quickly, or ways that that address it more thoroughly: possibly those things are even more awesome.

Because of this, I didn’t spend too much time locking my proposal down to the details of the exact thing I’m recommending. Instead, I wrote more about the types of functionality that I am proposing, and what customers have told me they want to be able to do quickly and easily: along with some notes about why I believe this is all humanly possible.

What Happens Next?

Redgate has a quite open culture. Earlier this week, I shared my proposal out with my team and a few coworkers. Yesterday, I shared it with the stakeholders for the products involved in my idea.

Today, I got a meeting invite to talk through things with the product teams next week.

So You’ve Got a Meeting: What If Nothing Happens?

As much as I like my idea, it’s not something that’s likely to happen overnight. It’s not a small amount of work, and there’s also other awesome things in the works.

So, even if my idea gets prioritized, I don’t consider my proposal to be “done.” I plan to add more customer use cases to it over time as I come across them, and also to regularly ask other people at Redgate if they have new ideas in this problem space that would make a big difference. Since the bulk of the proposal is written, this maintenance piece is not particularly hard to do, and it will keep my proposal fresh: just right for the exact moment when the stars align and it’s the right moment for this change.

This Isn’t the Only Way to Inspire Change

The approach that I described here is one that I’ve tailored to my situation and my audience. There are other ways to do this.

For some ideas, you may be able to create a working prototype or proof of concept of what you’re talking about: even potentially put it to work on a small scale. When that’s possible, you get even better customer stories and an ability to show the potential impact on dollar signs in another way. Absolutely take the option to “just get started” whenever you can.

Managing Cross-Database Dependencies in Builds (Redgate Video)

help@KendraLittle.com (Kendra Little) — Tue, 25 Jun 2019 00:00:00 +0000

Building your database code is an essential practice to ensure that it compiles from source and that dependencies are met. But things can get tricky when you have objects in some databases which is dependent upon objects in other databases: or even circular dependencies.

Video Overview

In this 20 minute video, I give an overview of the two most popular options that Redgate customers use to manage cross-database dependencies when building SQL Server databases with Redgate’s SQL Change Automation.

How Developers and DBAs Collaborate in a DevOps World (Video)

help@KendraLittle.com (Kendra Little) — Thu, 13 Jun 2019 00:00:00 +0000

You’re a DBA, and your development team is all-in on doing DevOps, and they want to include the database. Should your DBA team limit the permissions or options for automation? Or should you instead re-think how your two teams work together?

In this 40 minute episode, I discuss DevOps team topologies, the changing role of DBA teams in DevOps, and the opportunities available in creating communities of practice around database development.

🔥 LISTEN: Prefer to listen on the go? You can get this episode on iTunes, download the audio file, or find Dear SQL DBA in your favorite podcast app.

Bookmarks of Topics Covered

00:30 - Our question, from a DBA
7:00 - Why bother with DevOps?
9:23 - Team topologies, aka rethinking how the teams work together. This discussion includes notes on where and when DBAs review changes and talk about architecture in a DevOps way of working.
21:20 - Changing the DBA role from gatekeepers to internal consultants
32:30 - Creating a community of practice
38:38 - Recap and wrap up

Links from the Video

DevOps Topologies - https://web.devopstopologies.com/

Communities of Practice - https://wenger-trayner.com/resources/what-is-a-community-of-practice

A Letter to My 20 Year-Old Self

help@KendraLittle.com (Kendra Little) — Tue, 11 Jun 2019 00:00:00 +0000

This post is a part of #tsql2sday, a monthly community ritual where a topic is proposed by a community member and everyone is invited to join in.

This month’s topic is from Mohammad Darab, who encouraged us to: “Write your 20 year old self a letter. If you could go back in time and give yourself advice, what would it be?”

The advice I would give to my 20 year old self is the same advice I give to myself today, more than 20 years later:

You Are a Person Who Throws Yourself into Your Work: All the Way into Your Work

You tend to define yourself by your professional activities, and you expect your work to be consistently of very high quality.

One of your greatest strengths is that you are a creative person, but when you narrow your focus too much, when you make your life too much about work, and when you try to make things ‘perfect’, your creativity suffers. Life is a messy business, and to both be productive and to enjoy life, you need to get out of your “WORK ROBOT” mindset quite often.

For these reasons, it is important for you to set some boundaries on your work life, and to also put effort into tending to your life outside of work.

Tactically, this means that you need to practice:

Regularly turning away from work messaging and tasks when you are not at work and not on-call. That email can wait, no matter what that little voice in your head suggests.
Finding hobbies and activities outside of the workplace and showing up for them, regularly.
Staying in communication with your friends and family, and practicing getting back in touch with people you value when you drift apart.
Saying “no” at work in a multitude of ways, many of which don’t use the word “no” directly, but do involve you not doing everything yourself. You will feel guilty most times you do this, but that does not mean that the practice is wrong.
Asking for help at work in a multitude of ways when tasks can’t be completed as they were estimated.
Accepting that “good enough” is often, in fact, good enough. You will repeatedly be frustrated when you can’t make something perfect and when you make mistakes. Often, it is best to own up to the flaws in yourself and the situation and simply accept it, then move on to the next challenge where you may be able to contribute more. Look forward, girl, there’s cool stuff ahead.

On a Personal Level, Throughout the Years You Will Worry: You Will Worry a Lot

You will worry about being a nerd, and that you will never really be loved because you are too loud / too fat / too nerdy / too opinionated / not nice enough / not pretty enough / not clever enough / too ‘much’.

You will worry that not everyone likes you, and need to constantly remind yourself that it’s fine and right and normal for not everyone to like you, and that “being liked” is not your measure of success.

You will struggle to shore up your confidence, again and again.

You will frequently experience “Impostor Syndrome,” and to compensate for this you will tend to spend even more time working, even more time attempting to work on something so hard that it is beyond criticism.

There is no easy fix for over-worrying. Tending to your non-work life and helping that flourish will help: this wider perspective helps you to better find role models and sponsors, appreciate your strengths, find opportunities to help others, take criticism constructively, let non-constructive trolling roll away into the abyss of the internet, and to enjoy speaking in your own voice.

But there is no way to banish an over-worrying tendency forever. Learning to let your worries pass instead of clinging onto them like the world’s worst life raft will be a life-long project. You will constantly find new tools. You will always need to adapt.

The good news is that you do seem to love a good long project.

So get at it, friend. There’s still a lot of awesome things to do. Life passes in a flash, but while we are here, we always have the chance to appreciate it and to make it better.

DBAs: Stop Denying Sysadmin to Developers

help@KendraLittle.com (Kendra Little) — Thu, 30 May 2019 00:00:00 +0000

I recently chatted with some folks who have a permissions problem in SQL Server. The permissions problem isn’t technical: it’s a process problem.

The issue is that these folks are trying to configure a build for their SQL Server databases using Redgate tools, but they aren’t allowed sysadmin permissions on any SQL Server instance in their organization (even in development environments), because of a policy set by the Database Administrators in IT.

Why Do DBAs Deny Sysadmin Permissions in Development?

You may find this type of policy puzzling: and for good reason. After all, if you don’t trust developers to keep their own development environment running, why would you ever trust any code they’ve written enough to deploy it to production?

In that question is a bit of an answer: this policy occurs when there is a fundamental lack of trust between development and operations.

The policy that developers can’t have sysadmin rights in development has two main justifications used by database administrators:

Concerns that development environments will drift so far from production standards that they will introduce risks to code quality
Concerns that allowing any account sysadmin rights in development will lead to application accounts using sysadmin rights in production

Denying Sysadmin Rights to Developers Locks You into Manually Controlling Environment Drift: Instead of Managing It the Smart Way

If you’ve worked in IT or in developing changes for databases for a while, you’ve probably come across development database instances that are a complete mess: there are databases with odd names which maybe haven’t been used in a long time (but take up a lot of space), jobs failing, low disk space, and settings which resemble nothing in production. This messy type of development instance isn’t completely surprising, because developers need room to experiment.

Sometimes when the DBAs get called in to help support and instance like this, they respond by locking down permissions. If developers have to ask to create a database, we won’t get unexplained databases, and so on. The idea is to prevent the drift of these shared environments.

But treating developers like children won’t improve the quality of development long term. Removing permissions in development environments instead limits the ability of developers to experiment and automate, and limits the tools you can use to improve code quality.

How DBAs Should Fix a Drifted Shared Development Environment: Blow It Up

It’s true that nobody wants to support a shared development database environment. But here’s the thing: you shouldn’t even have that shared development environment. As Troy Hunt explains in “The unnecessary evil of the shared development database,” it makes a lot more sense for each developer to have a dedicated environment to work in.

One of the things that happens when you develop in a private environment and use standardized practices for your database code is that you automatically begin to control environment drift, by limiting routes for deployment.

In this world, there is no “wild west” shared development environment. Instead you have a workflow like this:

Each developer has a private development environment
Each developer checks their database code into source control
Branches in source control are combined with the private development environment to provide isolation / room to experiment for changes (in a way that will not impact other developers)
When code is ready to be merged in, automated processes build the code: proving that the code in source compiles properly and that all dependencies are met
- Having sysadmin rights for the process building your code is helpful for flexibility: for example, you might want your build to create a SQL Agent job if it does not exist, etc
Automation also provides support for deploying that code to a fresh environment for review

This process emphasizes that development environments need to be able to be quickly reset / recreated whenever needed, and that doing so should not impact other developers.

This approach also forces developers to ensure that configuration changes are standardized and are handled by code as much as possible, as there is no ability for them to do manual configuration to make the build or review process succeed.

Most importantly, this approach reduces environment drift while also using automation to improve code quality.

But a key to making this work is allowing high permission right to the accounts which are carrying out the build and other automation.

What About the Possibility of a “Sysadmin Free for All”?

The other reason this policy creeps up is the idea that developers can’t be trusted to follow the principle of least privilege, and if you give them sysadmin rights for anything they will use it for everything.

Let’s assume that’s a correct assumption for a moment. Giving developers lower privilege won’t actually fix the problem if they are always going to configure accounts to have the maximum permission allowed (“everything except sysadmin”).

This, again, is an issue of trust.

The better answer here is not to focus solely on what developer’s permissions are in the development environment. The answer is to focus on the permissions that application service accounts have, and to examine when and how code reviews can make sure that the roles used by service accounts are following the principle of least privilege.

To Survive, DBAs Need to Stop Being the TSA

We increasingly live in a world where the ability to deliver changes in software quickly, without impacting the user experience, is a core requirement to doing business.

However, we still live in a world where database administrator groups create policies that slow down development.

Worse, these policies also prevent the effective use of tools: like automation for builds and deployment of code to fresh environments for review, that improve code quality and reduce the risk of environment drift.

If you were a business owner and were investigating how to speed up time to market for your products, would you want your DBA team to be creating and enforcing policies that slow down releases and don’t improve quality?

Or would you be tempted to replace them with a different team, who would invest in using automation to improve code quality as well as improve release tempo?

Don’t make your organization choose. As DBAs we need to leave our obsession with policies behind, and focus on how to get what we truly want: high quality deployments and efficient management of our environments.

The Best Bit About SQL Saturdays: Nearly Everyone Interacts with Other People

help@KendraLittle.com (Kendra Little) — Tue, 21 May 2019 00:00:00 +0000

Today I was looped in on an email thread about the pros and cons of attending a specific event. One person on the thread asked if any of us had attended the event in the past, and whether or not event attendees were engaged with presenters and vendor representatives.

My immediate thought was: of course the attendees were engaged, because the event is a SQL Saturday. I’ve never been to a SQL Saturday where the attendees weren’t engaged.

But, I realized that it’s a fair question.

This Level of Interaction Isn’t a Given for Events, Much Less Tech Events

I’ve been to other tech events: from free events to very expensive ones, where attendees didn’t interact much with anyone else. Mostly people worked away on their laptops/phones/tablets quietly (maybe taking notes, maybe playing Minecraft, hard to say). The speakers seemed a little inaccessible and like they were very busy apart from when they were speaking, and they’d generally disappear right after their talk.

There are definitely a lot of people who attend SQL Saturday events who aren’t yet part of the broader SQL Server / Microsoft Data Platform community. They’re at a crossroads in their career and they’re not completely sure if this technology is for them yet, and they’re checking out this free event to see what they might learn from it. But I think the unusual thing about SQL Saturdays is that these events tend to draw these new folks into conversation with people who are already part of the community.

SQL Saturdays Are Very Casual, and You Don’t Need to Act Like an Expert

It’s traditional for speakers at SQL Saturdays to encourage questions and comments, and usually at SQL Saturdays there is no lack of attendee questions.

At some tech events, nobody ever “breaks the ice” during sessions by asking a question that’s accessible to beginners, and it can be tough to get the audience to ask questions or make comments. (Aside: if you are a speaker and would like me to ask a dumb question* from the audience to break the ice in your session, I am happy to do so! I am proud to provide this free service anytime I am around.) I’ve never found “too much quiet” to be a problem with SQL Saturdays, even at the events in countries where people are traditionally more reserved. I attended a lunchtime session at a SQL Saturday in Cambridge where Grant Fritchey got a room full of British people to share their experiences and pain points throughout the whole hour.

SQL Saturday organizers are also very good at packing their schedules with a variety of sessions that are quite accessible for people at different phases of their career and different experience levels, which helps everyone find content they can relate to.

🔥 On dumb questions: what I generally mean by a "dumb question" is one that makes the asker sound like a bit of a newbie / beginner / person who isn't in-the-know. I have become a person who loves to ask these, because having the question asked and answered thoughtfully often changes the dynamic in the room and makes it OK for everyone else to be a bit more vulnerable.

A Huge Factor Is That People in the SQL Saturday Community Are Extremely Welcoming

There’s a certain community vibe at the event that I think newcomers often feel and want to be part of. You see people who are chatting about what they’re up to, their experiences, and what they want to learn, and it’s natural to want to be a part of that.

I tend to take this community vibe, and in fact the supportiveness of the whole SQL Server community for granted, most of the time. But it’s a real thing, and it’s great to be a part of.

So, What Are You Waiting For?

If you haven’t been to a SQL Saturday and you’re reading this, check out SQLSaturday.com and look for SQL Saturday events in your area or your region. Even if you’re not committed to working in the Microsoft data platform. Even if you’re just a little bit curious about learning something for free on a Saturday. It’s a wonderful event, and it makes it easy for you to meet new people, to check out all sorts of new things, and to get engaged with the community of people involved in the Microsoft Data Platform.

I hope to see you at an event soon.

How to Recognize the Early Stages of Burnout, and My Treatment Plan

help@KendraLittle.com (Kendra Little) — Tue, 14 May 2019 00:00:00 +0000

I recently realized that I’m in the early stages of burnout.

This isn’t an unfamiliar place for me, but it is new for me to recognize the early signs of burnout in myself before it becomes a full-fledged disaster. This time, I’m thinking about how I got here, and making an explicit plan to change course.

In hope of helping someone else out there, I thought some public journaling might be in order.

How Do You Recognize If You’re in the Early Stages of Burnout?

I have recognized two symptoms which I identify as unusual for me. Together they indicate I’m heading towards burnout.

Symptom 1: Lately, I get frustrated and angry by small things

One warning sign of burnout is when small inconveniences start causing a disproportionately large emotional response.

For example, on a recent weekend I was traveling for work. I was in the Detroit area, about to head to London between a client visit and a conference. It was a beautiful day. I stopped by a local Starbucks and, by chance, another customer was rude to me.

Normally, if I wasn’t in the early stages of burnout, I would assume the other person was having a bad day, and I’d shrug this off. It wouldn’t be something that I’d even be likely to remember. But in this case, I felt a lot of anger toward that person. I was livid.

I quickly realized that the way I was feeling was more about me than about that random person in the Starbucks, but it took me a long time to shake off the anger.

This aspect of burnout is particularly tricky, because natural little miscommunications at work can slow you down more than normal, as you’re now having to handle not just the miscommunication, but also keep your own stress and irritability in check.

Symptom 2: Lack of excitement

I’m also just not as interested in work projects as normal. I’ve got a lot scheduled, but I often feel like I’m overwhelmed, and that all I can do is the bare minimum.

This lack of excitement contributes to:

Less curiosity and asking fewer questions
Taking less time to connect with my teammates and chat
Poorer listening skills
More hurried work / less critical thinking

Those things together mean that my work quality goes down a bit. And then that frustrates me.

What Causes Me to Burn Out?

Like I said, I’ve been here before: and I’ve been past this point, to where I simply couldn’t cope with the stress of my daily job anymore. Looking back, I can see some trends.

I have a tendency to enjoy working a little too much. And I have a few traits that I believe pave the way to burnout:

I tend to be a perfectionist, and I always want to help. I don’t like saying no, I always want to be involved when asked. I like finding a way to not only make things work, but to try to give them an interesting twist, too. I spend more hours working than I should: and by saying, “more hours than I should,” I mean…

I haven’t been giving myself enough time to recharge. For good health, I need to spend time away from a computer. I need to get exercise. I need to spend time outside. Meditating on a daily basis helps me ward off anxiety. Spending time with friends in person really helps as well. One of the things I realize now is that I haven’t been doing enough of these things lately.

I stretched myself thin and didn’t leave any room for life to happen. I signed up for a few more work projects than I should have this spring and summer. They are awesome projects, they are interesting and exciting and important. But I maxed out my schedule (plus a little), complete with loads of travel. It looked barely doable: until my dog, Mister, unexpectedly passed away. There was no room in my busy schedule for me to grieve my best doggo friend, and he wasn’t there to wag and say it’d all be great anymore. I started to feel trapped.

(( Perfectionist + Overcommitment ) - Self-Care ) × Grief = Tire Fire

As a career Ops person, I always want leeway. I like to plan a course where I’ve got a backup plan in my back pocket, and ideally a few viable alternatives behind that.

When you are headed towards burnout, you start feeling like there’s no leeway. No alternate plans. You have more stress than you can handle. You’ve got a seat on the struggle bus, and you’re not sure who is driving it.

So, what to do to change course?

Here’s my plan.

Step 1: Book time off

Yes, I am presently oversubscribed. But the very first thing I did when I recognized that I’m heading towards burnout was that I went into my calendar and started requesting days off wherever I could possibly make it work, or where someone else might be able to make it work in my absence.

This sounds counter-intuitive, but it’s necessary. While I can’t take a week or two weeks off right now, what I can do is:

Make a proposal for days off
Explain why I’m asking for those days off to my colleagues and boss, and ask for help to make it happen
Commit to not working during that time off: no notifications, no emails, nothing

Oddly enough, I find that it’s harder to disconnect from work than normal when I’m close to burnout. It’s something about the stress: it makes it harder to put work down. But disconnecting is truly needed, and for me I think this is one of the biggest ways to avoid burnout.

Putting work on pause and having time off helps give needed perspective on life. It reduces stress, eases the tension causing those knee jerk reactions.

It will also pay itself back by helping make me more focused and efficient when I am at work.

Step 2: Spend time with humans (of the non-work variety)

The second thing I did after recognizing the symptoms of burnout was to email my friends at home and suggest getting together. I was lucky to have a good friend who had just started up a conversation about this, but I looked around at my other friendships and thought about others who I haven’t seen in a while, as well.

As an adult, it can be tough to make and maintain friendships. But the time and effort is so worth it. For me, it makes me a happier person, and that happiness extends into my work life.

Just like disconnecting from work, shifting gears and making time for friendships can be a mentally tough thing when you’re feeling burnt out. My mind tends to fixate on the problems at work, and it wants to stay there.

But I know from experience that planning a hike with a buddy is so much better for me in the long run, so a big part of my “anti-burnout” plan is making sure that I’m getting at least four hours of non-nerd-time human contact a week for the next few months. (I know, it’s just so “perfectionist” of me to set an hourly goal, right? I like specific targets.)

Step 3: Pick an anti-anxiety habit (or two)

For me, daily meditation is very helpful. I have learned in the past that this is a simple tool that is quite effective at reducing my stress and anxiety. When I start doing this on a daily basis, it has a more significant effect each day.

I’m starting this slowly at just five minutes a day of meditation. At first my whole goal is simply to re-establish this as a habit, without mentally scolding myself if I skip a day. The point is to keep starting until it becomes something that I look forward to each day and it is once again natural.

I find that when I do practice meditation, I have a more balanced view of things and I am more able to ask for help. I’m also better at thinking of alternatives for how something could work when someone has a request that I can’t fulfill due to time commitments.

Journaling is also helpful for me. I’ve started writing for this blog in a new way. I’m finding that it’s helpful in a similar way that journaling helps me think through things.

For example, I dictated the first draft of this blog post aloud while walking around a hotel room looking out the window. An app on my phone recorded the audio and uploaded it to the cloud. Another app created a transcript, which I edited for the post.

This method encourages me to be more conversational and more personal in my writing. That’s very very helpful for me right now, because it’s a little bit more about my experiences and it’s a little bit more about getting my thoughts out in a way that is helpful and therapeutic for me. It also just makes me more excited about writing again, which is incredibly welcome.

What if I don’t have time?

The voice of burnout in your head is likely to make an objection: we don’t have time for this. That’s the whole point.

Well, here’s the thing that I’ve learned from the past: it may hurt to make time, but it’s going to hurt even more if you don’t.

It’s not easy to ask for help with your workload. You may need to negotiate to make it happen. It’s not a good feeling to say that you can’t do things which you’ve agreed to. If you’re in a culture of over-achievers, it can be quite difficult to say that you don’t want to work as much as everyone else is working.

However, the thing about burnout is that you can’t sustain it. If you don’t take action and you just keep your nose to the grindstone, chances are good that you’ll become desperate for a job change, and that you’ll either quit your job or take something, anything, for a change.

Burnout leads to bad choices.

It’s a much better choice to start doing the tough work and speak up for your own needs, before you are so burned out that you can’t. Get yourself into a more productive place before making any big decisions about the future: and things will look better from your new vantage point.

Paying Down Technical Debt

help@KendraLittle.com (Kendra Little) — Tue, 07 May 2019 00:00:00 +0000

One of the cool things that I do as an Evangelist at Redgate is to periodically visit company headquarters in Cambridge.

The other Evangelists and I get to meet with every software developer, product manager, and UX designer at Redgate over a series of meetings. We talk about features that teams have released lately and what teams are looking at doing in the near future. We get to give feedback based on what we hear from the community and from folks in the sales process. We also get to share what we think teams should work on.

As You Might Imagine, I Have a Wish List for Features in a Variety of Different Redgate Products

Our products are great, and one of the things about great products is that users are always inspired to want to use them in new ways, so I never lack for ideas.

So, I have a lot of opinions about things that I think should happen, and features that I would love to have for customers. And, of course, I’d like those features right now, please.

In a Recent Meeting with One of the Teams, They Mentioned That Most of Their Work Over the Next Couple of Sprints Involves Working on Their Continuous Integration Process

This sounds like a bummer, right? It’s a time period when the features on my wishlist, and Steve’s wishlist, and Grant and Kathi’s wish lists aren’t getting worked on.

But what the team explained was that this application has been around for a while, and a large amount of tests have accumulated. It currently takes more than 10 hours for the build and test process to run. There may be duplicate work going on in the tests, and there are probably tasks that can be made much more efficient. The long build and test time currently makes it painfully slow for the team to iterate on developing and testing new features.

The team said that in the long run it’s worth paying down some debt and making the automated build and testing cycle more efficient, so that they can iterate on features faster in the future, instead of having to find other things to do while waiting for the CI process to complete.

This news wasn’t greeted with cheers from all the Evangelists present: but, to be fair, when we do respond to something with cheers it makes some of the teams look at us oddly, as we’re in the UK and that’s not something they see every day at work. (Hey, I bring my American enthusiasm everywhere!)

But everyone in the room agreed that speeding up the build and test cycle as much as possible is a necessary and reasonable thing to do

Like any other set of users, we want what we want (and we want it ASAP), but we respect that to make software development work well, you occasionally have to step back and pay down some technical debt.

This Is Also True for Database Development

It’s not always obvious that when doing DevOps, stability is just as important as release frequency, but that is the case, and maintaining that stability requires being diligent about tidying up one’s processes.

As database professionals, as developers and DBAs, I believe what we should see as the ideal release cycle is one in which we are free to release features every day without any manual work, and we have a software development cycle in place that ensures that the risk of our changes is minimized, that coding patterns are being used that ensure system stability, and that we have a response process in place that restores services as quickly as needed should there be a performance, availability, or functionality problem.

But that doesn’t mean that we actually release changes every single day. In order to do that effectively, we usually have to have paid down a lot of technical debt. That means stepping back periodically and working on improving our processes, rather than relentlessly focusing on shipping, shipping, shipping, shipping.

This Is Even More Critical with Legacy Applications, Where There Is a Significant Amount of Technical Debt to Pay Down

Spending release cycles on making continuous integration and continuous delivery/deployment work better isn’t the part of DevOps that gets business owners and users really excited. But it’s still important to talk about, because this is a critical activity that enables us to deliver value on a regular basis: and that is what gets those folks excited.

What Is Automation in the Software Development Life Cycle?

help@KendraLittle.com (Kendra Little) — Mon, 29 Apr 2019 00:00:00 +0000

Today I got a bit closer to a meaningful definition of automation, as it applies to the software development process. I’ve been turning this concept over in my head for a while, which is partly related to the dreaded question of licensing.

A few weeks ago, I was chatting a bit in the SQL Server Community Slack Channel. One community member was frustrated with running into situations with per-user licensing for monitoring and automation products.

🔥 On the SQL Server Community Slack channel: The SQL Server Community Slack channel is great, join up here.

This isn’t the first time I’ve heard grumbling about per-user licensing, of course: with any licensing model, you’re going to hear grumbling about it, that’s just how licensing goes.

But I think per-user licensing can make a lot of sense when it comes to automation products, because of the nature of automation. I work for Redgate, which does per-user licensing. I also often do demos of how our tools integrate with Microsoft’s Azure DevOps Services (formerly VSTS / or TFS-in-the-cloud), which does licensing based on user numbers.

But not everyone thinks this makes sense.

That’s because they see automation as:

Something that one person sets up on a server, which that person may occasionally tweak; and…
A script or orchestrated set of scripts and products that replace the work that people (maybe more people than the person who set it up) would do manually

This definition isn’t dumb or naive at all. This is classically what automation has been in IT for many years: I’ve got a problem. I create a script. The script helps save me and my team some time and I only ever look at it again if it stops working.

Based on that definition, it would seem most natural way to be charged for automation tools would be based on something like the number of times the tools are run, the number of servers/cores they are run on, etc.

The Nature of Automation Has Changed Dramatically in Recent Years

Like I said, I’ve been having a hard time putting a definition of what automation means now into words. Then I saw a link to this job description for a Sr. Resilience Engineering Advocate at Netflix.

There are a lot of interesting things about the job description, but one sentence that leapt off the page to me was that the team values:

Automation as a team player versus automation as a replacement for humans

Netflix Cloud and Platform Engineering SRE Team

This is a huge part of the evolving definition of automation. Automation is now:

Something that a team configures, interacts with, and improves on a daily basis
A script or orchestrated set of scripts and products that are an integral part of the productivity of the team

The big reason that per-user licensing makes logical sense to me when it comes to tools that are designed to be a key part of the software development life cycle is that the tools are meant to be experimented with freely. The tools will work best if they’re able to be tinkered with and adapted over time, to suit the needs of the team at that point. Licensing based on cores or CPU cycles or usage naturally reduces experimentation if it is going to drive up cost.

Also, the tools are meant to be team players: they are meant to be available to have every team member interact with them. Automation in the SDLC for database changes doesn’t mean that every time a change is committed, the change rockets toward production without a human being ever needing to think about it again. Instead, automation is a player in a process that can absolutely include rigorous review (both automated and human-powered), testing, and even approval gates when needed.

Automation Looks Different in Different Teams

One observation: team size matters. If you’re one person in a small shop and you’re setting up automation to reduce the amount of manual work that you personally have to do, this high-faultin’ definition of automation as a “team player” probably isn’t going to resonate with you. You’re much more likely to continue to see automation in the classically defined sense.

But, on the other hand, you don’t have to have a team nearly as large as Netflix to start seeing the advantages of thinking about automation differently. It just takes a few people working together collaboratively and thinking about how to more consistently and reliably deliver values to customers to start changing the way automation exists in the workplace.

🔥 On licensing costs: I don't mean to make this post about how much software should cost. I actually don't think that's too terribly related to licensing model choice at all: whatever you are charging by, whether it be people, cores, tentacles, or whatnot, you can find a way to make it cheaper or more expensive.

Should Release Cadence Be Slowed If You Don't Have Database Load Testing?

help@KendraLittle.com (Kendra Little) — Tue, 23 Apr 2019 00:00:00 +0000

I got a question recently about a panel discussion on Database Development Disasters at SQL in the City Streamed. I had framed a question as, “how fast should development go without load or performance testing?”

I got a follow-up question from my friend Chris Randvere at Redgate: he asked for more information about what the question meant? I realized that my wording had been pretty unclear. I had meant to ask the panelists what their thoughts were on release cadence when a team lacks tooling to do automated load and performance testing outside of production.

Should the Lack of Automated Performance Testing Ability Change the Rate at Which We Deploy Software?

In other words, if we can’t do performance and load testing, does that mean that we should or shouldn’t deploy a change to a database every weekday?

I don’t think we covered this super-well in the panel because I worded the question poorly. So I wanted to share my experience around this, and also talk about why it can be fairly common for teams to lack automated load testing ability outside of production.

Why Doesn’t Everyone Have an Environment Where They Can Validate Performance by Replaying Activity Against an Updated Database Before It Ever Gets Released?

We have some built-in tools for this in SQL Server. The current version of this is called Distributed Replay. These tools are not the most lovingly tended by Microsoft in terms of updates. A frequent complaint that I’ve heard about distributed replay is that the current version of the tooling still requires you to feed it with old style profile traces done with the old SQL Trace.

You don’t necessarily have to have the Profiler app running while you do the trace, but the old-style SQL Trace results are what it takes in in other words. You can’t do a more modern Extended Events trace and feed that into the tool.

But that lack of updates isn’t the main reason why not everyone runs Distributed Replay.

Distributed Replay Is Tricky to Set Up

The more complex your environment, the trickier it is to set it up. If you’ve got things like linked servers or SQL Server transactional replication creating interesting patterns in which your SQL Server communicates with other SQL Servers, that can make using Distributed Replay particularly tricky.

There are absolutely people out there who’ve configured Distributed Replay on complex systems, but they all say it wasn’t something they set up in just two hours. So one factor is the complexity.

Another Factor: Distributed Replay Is Designed to Replay, Not to Amplify

When we’re doing performance or load testing, we are not always interested in: how will the system perform under the current load? A lot of times we’re interested in: how will the system perform if it’s under even more load? 150% of the load, or 200%, or more.

But with any replay tool: I’m not just dogging on distributed replay here, if we think about the nature of any database replay tool, we can’t replay the exact same commands and expect to learn how performance will be at a higher load rate.

For example, a delete command. If a delete command on the first run finds 10,000 rows to delete, that could be a fair amount of work.

If we replay that same delete command, depending on what the criteria are in the delete, it possibly will find zero rows to delete the second time, because it’s already completed. Similar things can happen with updates. We may also have constraints that mean we can’t just insert the same thing twice depending on the nature of the data.

So the way modifications work, just amping up the load in a replay isn’t the same as adding true additional load.

Now, there are other ways you can do load testing. There are third-party tools that you can buy to get around this problem of repeated modifications.

They can be expensive. But they also require a fair amount of coding, because you’ve got to put in commands that help you get to a state where you can check: Hey, let’s turn let’s turn the volume of activity up to 200%, to 300%.

So, some folks do that. but because of the cost and the effort put into it, folks only tend to do this with third party tools when it’s really worth their while, and their management is deeply invested in the idea of having load testing.

Even Then, the Load Testing Needs to Be Updated for Some Changes

For some changes, you can test them with load testing tools without any changes. For example, if I refactor a function for performance tuning, but don’t change any inputs or outputs, I could test that with an existing configuration of a load testing tool.

But what if I add a new parameter onto a stored procedure? If I don’t change the load testing, will that be appropriate, or not? Should I be running the load test with a variety of values for that?

Or what if my change involves dropping one procedure and adding in another one? A replay system would have no idea what to do, and with a load testing system I’d need to adjust what gets executed, how often, etc.

When it comes to testing database changes, load testing tools are excellent, but I’d expect some human work to be required as well.

So, Most People Don’t Have Automated Performance and Load Testing: Should That Impact How Frequently We Deploy Changes to Production?

What we’re really looking to find with load testing in the SDLC cycle is regressions in performance. We should have other testing to catch true defects such as making sure that the right results are returned, etc.

With database changes, there is a fair amount of work we can do to make sure that things perform well without load testing. There is other due diligence that can help: we can maintain and use a staging or pre-production environment with production size datasets and the same data distribution as production, for example.

In that environment, we can testing that confirms: what indexes is the modified code using, and are they optimal for the modified code? How long are queries taking? We can make educated guesses about production performance instead of waiting until after a release to see what it’s like.

This level of manual performance testing can work extremely well, and it unlocks us to do frequent deployments, in my experience.

Without Load Testing, It’s Best to Frequently Deploy Small Database Changes

By small, I mean as small as possible. A lot of these database changes are going to be things that our customers shouldn’t notice at all. Like, hey, we add a new column to this table. We’re not actually using it, though. We’re going to use it in the future for a feature that’s coming out soon.

But we regularly trickle out the staging steps for a future change. Each of these steps is backwards compatible and deployed well ahead of the point in which we “turn on” the new feature for our customers, which is often handled via an application control called a feature flag.

This regular stream of very small changes is helpful for speedily resolving any performance issues which may occur.

If performance does change, we haven’t released a big batch of 50 changes all at once: that’s a lot of things to go through to find the culprit. Instead, we’ve released maybe 7-10 changes in the last week, and we can look at the most recent ones first, and check if they could be related to the issue.

What Do You Mean by “Frequent”?

By “frequent”, I mean that daily releases shouldn’t be a big deal.

But remember: batch size is critical. This isn’t like a race car, this is like a steadily dripping faucet.

Complexity: Sometimes Performance Problems Don’t Happen Right After a Change Is Released

There can be changes that we make where performance is fine for a while, and then something happens. Maybe it’s even three weeks after the change was released, but suddenly performance is terrible.

One particular tricky issue in SQL Server that can cause this is what’s called parameter sniffing. If we are using queries that are parameterized, a lot of the way that things perform depends on the execution plan that is cached with the first set of parameters that are passed into the query or procedure when it’s compiled, because SQL Server reuses that execution plan until something causes it to recompile.

Maybe the thing that causes the query to recompile is enough data has changed that statistics automatically update. Maybe it’s that there’s a failover in an Availability Group. Maybe the SQL Server is restarted. Maybe someone manually clears the execution plan cache.

A very wide variety of things can change, but if we happen to we have a recompile and we happen to have a compilation with a set of parameters that leads to an execution plan that doesn’t work so well, then we can suddenly run into performance issues.

But this risk of parameter sniffing doesn’t mean that we shouldn’t release that changes frequently. We’re going to have the same risk whether or not we’re releasing a big batch of changes or whether we are regularly releasing small changes.

For issues like this, I like the recent auto-tuning features in SQL Server. Essentially the automatic plan correction feature will look cases where a query is sometimes fast and sometimes slow, and attempt to identify “bad” execution plans which are periodically cached for a query.

You have to have the built-in Query Store feature enabled, and when using on-prem SQL Server this feature requires Enterprise Edition for production (Developer Edition has all the features of Enterprise for non-production environments). But if you do have a SQL Server where performance is critical, this feature is a huge deal. It gives you the option to either let it temporarily correct the plan for you, or to notify you in a DMV that it’s spotted a problem that needs tuning.

This feature can either work with automated load testing: you can use the feature to spot parameter sniffing problems before deployment, or it can give you early warning of performance problems before your customers start complaining. It also gives you insight into how to reproduce these tricky problems outside of production.

A Quick Recap

Automated load testing is fantastic, if you have the time and budget to take it on for your team.

But if you don’t have it, the lack of automated load testing shouldn’t block you from frequently deploying small changes to production. By combining targeted manual performance testing into properly planned changes, you can still deploy frequent changes safely into high performance and high uptime environments successfully.

Upcoming Full Day Training: How to Architect Successful Database Changes

help@KendraLittle.com (Kendra Little) — Fri, 19 Apr 2019 00:00:00 +0000

I’m excited to be teaching a full day session with Steve Jones at the SQL PASS Summit on Tuesday, November 5, in Seattle.

Steve and I will be discussing proven patterns to version and deploy changes successfully.

Read more about this precon session, or check out the video below where I give a brief overview of what Steve and I will cover.

Will You Teach Me How to Use Redgate Tools?

Nope: not in this session.

While Steve and I both work for Redgate, we will be showing patterns and approaches that work with both vendor and custom tooling, and we’ll do demos with a variety of tools, including free tools when possible. This is absolutely not a product-specific session, and the patterns discussed have been proven in the industry by developers and DBAs using a wide variety of tooling.

If you’d like to learn more about Redgate tools, check out the Redgate SQL in the City Summit Precon that will be held on Monday, November 4. That day of training has 100% different content, so if you’d like to go all-in on DevOps, join us for both!

Reserve Your Spot

You can currently register for an individual pre-conference session for $499, or bundle pre-conference sessions with the conference.

Hope to see you at PASS Summit!

The 2019 State of SQL Server Monitoring Survey Is Open: Here's What's in It for You

help@KendraLittle.com (Kendra Little) — Tue, 19 Mar 2019 00:00:00 +0000

Calling all Database Administrators, Developers, Analysts, Consultants, and Managers: Redgate has a survey open asking how you monitor your SQL Servers.

Take the survey before April 5, 2019.

Your time is valuable. The survey will take 5 - 10 minutes to complete. That’s not a ton of time, but it’s a noticeable part of your day, and there should be something in it for you. Here’s why it’s worthwhile to take the survey.

Database Use Patterns and Monitoring Trends Are Valuable to Everyone in the Community: and We’ve Been Missing Out on This Trend Information!

This isn’t the only survey from Redgate: you may also have heard of the annual State of Database DevOps Report.

But this survey is different in important ways: it asks how you manage and monitor your SQL Server database environment, regardless of whether you think about DevOps at all.

Redgate shared the results of the State of SQL Server Monitoring Survey with readers last year, and will do so again in 2019. 2018 was the first time we know of that a survey of this kind had been done, and it will be especially interesting to notice what has changed in the past year.

If you fill out the 2019 State of SQL Server Monitoring survey, you’ll get an advance copy of the results by email

The results will help you understand:

What do other DBAs, Developers, and IT Professionals see as their biggest challenges over the coming year?

In last year’s report the biggest challenges were seen as:

Migrating to the cloud
How to deploy changes faster to larger environments
Protecting data – especially for compliance reasons

I’m curious to see if these priorities have changed after a year. With GDPR implementation having come to pass and more states and countries around the world passing increasing privacy regulations, I suspect that ‘protecting data’ may move up from number 3 on the list, but I won’t know until we see the data.

How much time do your peers spend examining SQL Server health and resolving issues?

This is a great question on the survey: and I think this one is absolutely worth some reflection after you take the survey (and after the results come out). Do you spend more time firefighting than you should? If so, what ideas you have to change that?

At what point to most organizations move from manual monitoring to a third party tool?

If you work for a growing company and are interested in making the case to purchase monitoring to your management, it may be useful to know information like this…

Respondents with fewer than 10 servers were twice as likely to rely on manual monitoring as to use a paid-for tool. Those with 10 or more servers were more likely to use third-party software.

2018 State of SQL Server Monitoring Report

Survey Results Will Help You Prioritize What to Learn

Are you curious as to whether you should learn another database platform, such as MongoDB, Oracle, MySQL, Cosmos DB, or PostGres? The survey will show how much respondents report that they are using each platform, and whether they think the amount will increase or decrease.

Want to know whether you should invest time (and maybe ask for some budget) to explore cloud technologies like Azure Managed Instances, Azure SQL Database, or Amazon RDS? The survey will show how many of your peers are using each one.

Also, You Could Win Money

Everyone who completes the survey and provides their details at the end will be entered into a prize draw to win a $250 Amazon Voucher (or equivalent in your local currency).

Take the 2019 State of SQL Server Monitoring Survey

The SQL Server Data Row Size Question: Why Is It Bigger?

help@KendraLittle.com (Kendra Little) — Tue, 19 Mar 2019 00:00:00 +0000

This morning, I received the following question from a user:

Could you please clarify SQLServer “Data Row” size: If I run the script below on SQL Server 2012, then Slot(row) Size is 710 bytes if I run the same script against SQL Server 2016 and above, then Slot(row) Size is 724 bytes.

They then provided a script which creates and inserts a few rows into a sample table, runs the DBCC IND command to find a list of pages for the sample table, then uses the DBCC PAGE command to examine the page.

The First Thing I Looked For: How Many Bytes Is the Difference? Is It 14 Bytes?

I love it when someone sends me a repro script, but in this case I didn’t need to run it. The first thing I did was to look at the two numbers given for row size, and to subtract the smaller one from the larger one: 724 - 710 = 14 bytes of difference.

That bit of information alone gave me an immediate guess of what was going on.

Row Versioning in SQL Server Has a 14-Byte Row Overhead

You can reduce blocking in SQL Server by enabling “optimistic” locking. There are trade-offs for this approach, however: row versions are enabled in tempdb, and 14 bytes of overhead may be required on impacted rows. (For a quick overview of this row versioning process, check out this Simple Talk post by Kalen Delaney.)

My guess is that row versioning is enabled only on the SQL Server 2016 and above instances that are being tested in this case. This could be because of any of the following

Snapshot isolation is allowed on the database
Read committed snapshot isolation is enabled on the database
The database has a readable secondary in an availability group

You Might Wonder: Why Would Having a Readable Secondary Cause the Row to Grow?

Readable secondaries in SQL Server allow you to offload read workloads.

It’s important to minimize blocking against those readable secondaries, though: you don’t want data modifications flowing through to the readable secondary to be blocked by reporting queries that you are running against the secondary, after all. You also want reads to be as consistent as possible.

For this reason, queries that are run against readable secondaries are automatically escalated to snapshot isolation. And snapshot isolation requires row versioning – those 14 bytes of space are needed too help make it all work.

But readable secondaries are read-only, right? So how can it write those 14 bytes? The answer is that it’s complicated. Here’s the documentation from in the “capacity planning” concerns in Microsoft’s docs:

When you configure read-access for one or more secondary replicas, the primary databases add 14 bytes of overhead on deleted, modified, or inserted data rows to store pointers to row versions on the secondary databases for disk-based tables. This 14-byte overhead is carried over to the secondary databases. As the 14-byte overhead is added to data rows, page splits might occur.

The row version data is not generated by the primary databases. Instead, the secondary databases generate the row versions. However, row versioning increases data storage in both the primary and secondary databases.

Offload read-only workload to secondary replica of an Always On availability group

To dig into the details of this more, click the link – there’s even more info on the full page.

Why You Shouldn't Hardcode the Current Database Name in Your Views, Functions, and Stored Procedures

help@KendraLittle.com (Kendra Little) — Mon, 18 Mar 2019 00:00:00 +0000

I recently published Why You Shouldn’t Hardcode the Current Database Name in Your Views, Functions, and Stored Procedures over on Simple Talk.

What the Article Covers

In the article, I discuss:

Why referencing the current database name creates a dependency
What ‘deferred name resolution’ is, and why the dependency may be more noticeable in views and some functions (rather than stored procedures)
Which activities are most likely to break if you place dependencies on the current database name

Read the full article here.

PASS Summit 2019 Call for Speakers Is Open: How I Sketch Out My Submission Ideas

help@KendraLittle.com (Kendra Little) — Wed, 13 Mar 2019 00:00:00 +0000

Are you interested in speaking at the Professional Association for SQL Server’s annual Summit conference?

The call for speakers is now open, and you may submit up to three sessions between now and March 31, 2019.

I’m sketching out my ideas for what sessions that I’d like to submit, and I thought I’d share my process here.

Generating Ideas: What Am I Interested in Spending Six Months Thinking About?

I’m a bit selfish when it comes to topic selection, and I think that’s fine: it needs to be something that I’m interested in thinking about for more than half a year.

That does NOT mean that it needs to be super-advanced rocket science content. Figuring out how to present introductory level content clearly, in an easy-to-understand way takes a lot of time. The topic simply needs to compelling enough for me to stay interested.

A Filter: Is the Topic Relevant to Enough People?

When I first began speaking, I thought I needed to speak on topics that were unique to me, which other people in the community didn’t already “have covered.” This led me to somewhat esoteric concepts. There’s a big downside to that: your talks simply won’t be relevant to many people.

Now, I encourage myself (and you) to use the opposite filter: think about talks that will be be helpful to a lot of people. That means you probably won’t be the first one in the world to deliver a talk on the topic, and that’s perfectly fine: your perspective is valuable!

There is definitely a “three bears” aspect to this filter. Your talk doesn’t need to be useful to everyone at the event. But do think about your intended audience, and whether you’ll help a significant portion of the audience at the event.

Sketching Out Ideas and Audiences

For this year’s PASS Summit talks, I currently have two topics I’m quite passionate about, which I’d love to share.

Right now I don’t have abstracts. I’ve started by creating notes on:

Subject matter area / rough title
What the talk would include - a problem summary (which I write as quickly as possible, just throwing out ideas) and a bulleted list of content ideas
Who would care (audience) and why they would care

Here’s where I’m at with my two topics.

Topic 1: Source Controlling Index Code in a Changing World

Alternate title: I prefer, “How to Standardize Index Code in a Changing World.” Not everyone may get that Standardize = Source Control, however – not sure if that’s simply how I think of it.

Problem summary: It’s critical to get database code standardized into source control to manage collaboration, store and version your organization’s intellectual property, and to track and audit what has happened in your database code. The database code for indexes, however, is increasingly difficult to standardize: new features in Azure SQL DB automate the process of tuning index schema. Single-tenant database architectures often standardize table schemas, but require customizations of indexes for performance in individual databases. And index schema often needs to “drift” in production as operations teams respond to critical performance problems. How do you adapt successfully to this chaos, yet still maintain your sanity by managing your database code in source control?

Topics to include:

Differences in source controlling indexes with state vs migrations approaches to database code
- Strategies for working around limitations on advanced index features with a state approach
How to manage drift that occurs with automatic indexing in Azure SQL Database
How to manage incidents where indexes are manually changed/drifted in production due to a critical need without going through a full release cycle
How to manage single-tenant environments where the indexes are customized in individual databases

Audience / who would care:

Developers and DBAs who have their databases in source control, but who struggle with “drift” in some areas
Developers and DBAs interested in getting databases into source control – sometimes thinking ahead about these more advanced issues at the beginning can help you learn fast
Architects thinking about using single-tenant designs

Topic 2: Best Practices for Branching Database Code in Git

Problem summary: There are quite a few discussions and patterns available online for branching application code, but special considerations apply when it comes to database code – and hardly anyone has written about this! This talk helps DBAs and developers design the simplest branching strategy that meets the needs of their organization. I will share key considerations for designing a branching strategy for database code, and we will discuss multiple popular branching models along with “fit notes” describing the strengths and weaknesses of each model.

Topics to include:

The overarching rule of “keep it simple”, but why this leads to different practices for small teams (1-2 devs checking in code) vs larger teams
State-first vs migrations-first and explain what that means for branching - a migrations approach that doesn’t include a state component will SUCK for merging … maybe don’t use the word SUCK in all caps in the final abstract tho
Why shared vs dedicated development databases is important when considering branching - why object locking is a necessarily evil of a shared model, so don’t do shared if you can avoid it (esp considering that Dev Edition is free for SQL Server, side-eye at Oracle)
What feature branches are and how to tell if you need them
What release branches are and how to tell if you need them
Pull requests
Rebasing
Branching diagrams for multiple popular models, and “fit notes” for the teams using them

Audience / who would care:

Me!!! Hahahaha, I find this topic fascinating, even if there were five people in the room I’d be thrilled (but I do think this would be quite popular with dev-minded folks, I don’t think it’s just me)
Developers and DBAs who want to get database code into source control
Developers and DBAs who have code in source control, and who struggle with:
- Managing concurrent changes to objects
- Building the codebase from scratch to a specific release / version
- Managing drift in different environments / comparing database versions to source versions

Next Steps: Mulling These Over

I feel pretty good about these topics so far: I think that I care enough about them to want to spend a good amount of time with them, and I think that enough people at the conference care about the problems discussed to make them worthwhile.

These two talks are a bit more focused than talks I’ve done in the past, in that they’re specific to managing code in source control. I’m strongly of the opinion that everyone should be managing their database code in source control, though, so I’m fine with the level of focus – source control should be the norm!

I am presently waiting a few days before writing and revising “real” abstracts. I’ve got some time before the deadline, and I find it’s good that I sit with a topic for a few days to see if a bright new idea occurs to me, or to see if I may want to go in a different direction.

First Time Speaker? Go Ahead and Submit!

Writing an abstract and choosing a title for a talk can be daunting. I personally find it easier to start with making notes about a potential talk like I’ve shown here, and then refining those notes. I am hoping that helps some potential speakers out there to get started.

Here’s that link to the call of speakers again – submit by March 31, 2019.

Blogs on KendraLittle.com

Bad News, DBAs, We Are All Developers Now

A Pivotal Hackathon

Help from Teammates

Benefits of Owning Changes End to End

AI for App Code: Following Patterns Works Extremely Well

Where AI is Terrible for SQL Code

Where AI is Fantastic for SQL Code

Code Review Makes You Better and Builds Collaboration

App Code Projects I’m Thinking About

What This Means for Your Career

Adaptive Joins and Memory Grants in SQL Server

What Are Adaptive Joins?

Understanding Hash Join Build and Probe Phases

All Adaptive Joins Are Born as Hash Joins

Memory Grant Implications

When to Optimize with Temp Tables and Dynamic SQL

Example Query with Adaptive Join

Useful Adaptive Join Query Plan Operator Properties

Limitations of Adaptive Joins

Enabling and Disabling Adaptive Joins

Learn More

Carrying Baggage Through Query Plans: Why Wide Queries Get Heavy

tldr

Why this is Often Inefficient

Let’s Work Through an Example

A Wide Query Enters the Chat

Tuning Option: Temp Table

Tuning Option: Derived Table

Common Pitfalls to Watch For

Lightening the Load

Testing SQL Server 2025 Resource Governor tempdb Limits with a Query that Spills a Terabyte

Meet our Test Query Whose Sort Spills a Terabyte of Data

SQL Server 2025 Resource Governor tempdb Space Governance

Configure Resource Governor for our Test

Step 1: Create a Resource Pool

Step 2: Create a Workload Group with tempdb Limits

Step 3: Create a Classifier Function

Step 4: Configure and Enable Resource Governor

Step 5: Verify the Configuration

Step 6: Run the Test Query in a Classified Session

Step 7. Query tempdb Usage and Violations from sys.dm_resource_governor_workload_groups

The total_tempdb_data_limit_violation_count column does not count the number of queries that failed

Cleanup: Remove Resource Governor Configuration

Did Resource Governance Help with tempdb Spills?

How to Trace Trigger Executions with Query Store and Extended Events

Setup: Enable and Clear Query Store

Setup: Create the Audit Table

Setup: Create the Trigger on Posts

Setup: Create Query Store History Capture Helper

Setup: Create Test Data Procedure

Run Test Procedure and Review Audit Table Rows Inserted by Trigger

Review Query Store Data

XEvents Event File Session on sqlserver.module_start

XEvents Event File Session on sqlserver.module_end

XEvents Event File Session on Stored Procedure Statement Completions

XEvents Histogram Session on sp_statement_completed

Run our Test Procedure Again, Then Query All Trace Files

Run Test Procedure

Query module_start Events

Query module_end Events

Query Completion Events - sp_statement_Completed for Trigger Statements

Query Aggregate sp_statement_completed Events

Query histogram Target for Statement Execution Counts

Extended Events Observations

Cleanup

What We Observed About Tracing Trigger Executions

Static Site Improvements: Scheduled Posts, Preview Builds, Robots Writing the CSS

Recap of My Site Setup

Component Upgrades

Preview Deployments

Local Development Workflow

Standardized Buttons and Forms

Mobile Improvements

Client-Side Search with Pagefind

GitHub Issues for Blog Post Ideas and Bugs

Automatic Issue Syncing

Standardizing Categories and Tags

Future-Dated Posts for Scheduling

Netlify Automated Builds

Step 7. Query tempdb Usage and Violations from `sys.dm_resource_governor_workload_groups`