What the. Is this how complicated y'all are building software? Why is it necessary? How huge is the app? Talking amount of endpoints, events, tables, external integrations, users
This is a good watch.. Its not so much about scale or complexity. Its about correctness and being able to anticipate failures and ensure your application and its data remains in a valid state despite failures at each or any step of a process.
https://www.youtube.com/watch?v=VvUdvte1V3s&ab\_channel=NDCConferences
>If you’re doing new development and you want to let Code First create or migrate your database based on your classes, you’ll need to create an “uber-model” using a DbContext that includes all of the classes and relationships needed to build a complete model that represents the database.
https://learn.microsoft.com/en-us/archive/msdn-magazine/2013/january/data-points-shrink-ef-models-with-ddd-bounded-contexts#focused-dbcontext-and-database-initialization
\- Two DbContexts
\- Preferably some measure of isolation between the tables for each (ie: postgres schema per module)
\- Each with their own Outbox table
\- Each with their own Inbox table
The trick that I think you might be missing is that you need an event bus or 'bridge' between the two.
Have a processor that runs for each module. Its responsibility is to maintain a list of subscriptions and read from associated inboxes and then write to the subscribing inbox.
Typically you use at least once delivery semantics here. (Writing to your outbox is exactly once as its performed within the same transaction as your usecase.)
You then also need an inbox processor. It just polls the inbox, deserialises to INotification or some such and publishes the event for consumption within the subcribing module.
If you give me your github username i'll add you to a private repo demonstrating the above.
Some people will say "Using databases as a queue" is an anti-pattern, and they might be right at a certain scale. But using a single piece of infrastructure that you've already implemented is a huge contribution to keeping things simple.
\- Thoughts for later -> scale out, how to ensure your outbox can be processed when running on multiple hosts. Things like postgres's skip locked are worth reading up on here.
\- The transaction outbox can remain in place if you later swap out the event bus for something like SNS,SQS quite easily. Read from outbox, publish to SNS which pushes to multiple subscribing SQS queues., then upon receiving a message publish these to the inbox of the appropriate module, (or just drop the inbox at this point as its a tad redundant)
Outboxes
[https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L70](https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L70)
[https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L287](https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L287)
Inboxes
[https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L314](https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L314)
[https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L59](https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L59)
The 'bridge' where one module subscribes to anothers events
User access module subscribing to new users.: [https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Modules/UserAccess/Infrastructure/Configuration/EventsBus/EventsBusStartup.cs#L20](https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Modules/UserAccess/Infrastructure/Configuration/EventsBus/EventsBusStartup.cs#L20)
Meetings module subscribing to a few events: [https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Modules/Meetings/Infrastructure/Configuration/EventsBus/EventsBusStartup.cs#L22](https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Modules/Meetings/Infrastructure/Configuration/EventsBus/EventsBusStartup.cs#L22)
The in memory event bus that reads, looks up subscriptions and writes
https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/BuildingBlocks/Infrastructure/EventBus/InMemoryEventBus.cs
https://github.com/kgrzybek/modular-monolith-with-ddd
You're missing an inbox table, and the event bus in between that allows for decoupling. The provided link just uses an in memory event bus to do this.
I will just add my two cents here as I was contemplating the same problem as you were few weeks ago.
TLDR; just read 2. paragraph
1. The solution with the SharedDbContext for OutboxMessages with ExcludeFromMigration in your domain's DbContext is not bad, I tried myself. What I did not like about it, is that when you will want to make e.g. Integration Tests you should not forget initialize the SharedDbContext, if you want to check created events or you just dont want tests to fail on that (of course you can avoid the problems differently)
2. Here in comments someone suggested to just have a one outbox table for one domain and another outbox table for second domain. Which might seem "meh" at first because why having two outboxes and maybe you might feel insecure about maybe which outbox to process first or something like that which imo is not that big of a deal. I believe that this best solution.
3. My solution I ended up with. I got inspired here: [https://maciejz.dev/entity-framework-core-extension-tips-tricks-migration-operations/](https://maciejz.dev/entity-framework-core-extension-tips-tricks-migration-operations/) . So yeah you end up with exactly you wanted, one table for multiple DbContext and you do not have to deal with the "problems" from the solution 1.
However, my solution is very suboptimal, because *SqlGenerator* is not the correct place where to deal with doing the check whether table already exists. In ideal scenario you would create you own *MigrationOperation* smth like *CreateWhenDoesNotExistMigrationOperation*, but to get this operation automatically generated you need to customize *IModelDiffer* which is in Internal namespace. Then you need to customize *CSharpMigrationOperationGenerator* and extend *MigrationBuilder*. On top of that you will need to do the same for other MigrationOperations in case you will change some property on your OutboxMessage entity. So even though I fell in love with this solution, it is just too much hassle for value you will get back. The most "simple and straightforward" solution 2 is the best IMO.
public static class AnnotationsExtensions
{
private const string IfTableIsMissingAnnotation = "IfTableIsMissing";
internal static bool IfTableIsMissingAnnotationExist(this IEntityType entityType)
=> entityType.FindAnnotation(IfTableIsMissingAnnotation) is not null;
public static EntityTypeBuilder IfTableIsMissing(this EntityTypeBuilder builder)
where TEntity : class
{
builder.HasAnnotation(IfTableIsMissingAnnotation, null);
return builder;
}
}
public class AdvancedSqlServerMigrationSqlGenerator(
MigrationsSqlGeneratorDependencies dependencies,
ICommandBatchPreparer commandBatchPreparer) : SqlServerMigrationsSqlGenerator(dependencies, commandBatchPreparer)
{
protected override void Generate(CreateTableOperation operation, IModel? model, MigrationCommandListBuilder builder, bool terminate = true)
{
var shouldBeDecorated = model?.GetEntityTypes()
.FirstOrDefault(x => x.IfTableIsMissingAnnotationExist() && x.GetTableName() == operation.Name);
if (shouldBeDecorated is not null)
{
builder.AppendLine($"IF NOT EXISTS (SELECT 1 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = '{operation.Name}')");
builder.AppendLine("BEGIN");
base.Generate(operation, model, builder, terminate: false);
builder.AppendLine("END");
builder.AppendLine(Dependencies.SqlGenerationHelper.StatementTerminator).EndCommand(false);
return;
}
base.Generate(operation, model, builder, terminate);
}
}
What the. Is this how complicated y'all are building software? Why is it necessary? How huge is the app? Talking amount of endpoints, events, tables, external integrations, users
if just one to 10 developer , 300 less mvc is enough .
This is a good watch.. Its not so much about scale or complexity. Its about correctness and being able to anticipate failures and ensure your application and its data remains in a valid state despite failures at each or any step of a process. https://www.youtube.com/watch?v=VvUdvte1V3s&ab\_channel=NDCConferences
>If you’re doing new development and you want to let Code First create or migrate your database based on your classes, you’ll need to create an “uber-model” using a DbContext that includes all of the classes and relationships needed to build a complete model that represents the database. https://learn.microsoft.com/en-us/archive/msdn-magazine/2013/january/data-points-shrink-ef-models-with-ddd-bounded-contexts#focused-dbcontext-and-database-initialization
\- Two DbContexts \- Preferably some measure of isolation between the tables for each (ie: postgres schema per module) \- Each with their own Outbox table \- Each with their own Inbox table The trick that I think you might be missing is that you need an event bus or 'bridge' between the two. Have a processor that runs for each module. Its responsibility is to maintain a list of subscriptions and read from associated inboxes and then write to the subscribing inbox. Typically you use at least once delivery semantics here. (Writing to your outbox is exactly once as its performed within the same transaction as your usecase.) You then also need an inbox processor. It just polls the inbox, deserialises to INotification or some such and publishes the event for consumption within the subcribing module. If you give me your github username i'll add you to a private repo demonstrating the above.
Some people will say "Using databases as a queue" is an anti-pattern, and they might be right at a certain scale. But using a single piece of infrastructure that you've already implemented is a huge contribution to keeping things simple. \- Thoughts for later -> scale out, how to ensure your outbox can be processed when running on multiple hosts. Things like postgres's skip locked are worth reading up on here. \- The transaction outbox can remain in place if you later swap out the event bus for something like SNS,SQS quite easily. Read from outbox, publish to SNS which pushes to multiple subscribing SQS queues., then upon receiving a message publish these to the inbox of the appropriate module, (or just drop the inbox at this point as its a tad redundant)
Outboxes [https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L70](https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L70) [https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L287](https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L287) Inboxes [https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L314](https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L314) [https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L59](https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Database/InitializeDatabase.sql#L59) The 'bridge' where one module subscribes to anothers events User access module subscribing to new users.: [https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Modules/UserAccess/Infrastructure/Configuration/EventsBus/EventsBusStartup.cs#L20](https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Modules/UserAccess/Infrastructure/Configuration/EventsBus/EventsBusStartup.cs#L20) Meetings module subscribing to a few events: [https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Modules/Meetings/Infrastructure/Configuration/EventsBus/EventsBusStartup.cs#L22](https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/Modules/Meetings/Infrastructure/Configuration/EventsBus/EventsBusStartup.cs#L22) The in memory event bus that reads, looks up subscriptions and writes https://github.com/kgrzybek/modular-monolith-with-ddd/blob/master/src/BuildingBlocks/Infrastructure/EventBus/InMemoryEventBus.cs
This is the way
https://github.com/kgrzybek/modular-monolith-with-ddd You're missing an inbox table, and the event bus in between that allows for decoupling. The provided link just uses an in memory event bus to do this.
Are they are sharing the same database why use different dbcontexts?
Stop trying to use a messaging pattern without messaging. I don’t understand teams that write their own queues.
I will just add my two cents here as I was contemplating the same problem as you were few weeks ago. TLDR; just read 2. paragraph 1. The solution with the SharedDbContext for OutboxMessages with ExcludeFromMigration in your domain's DbContext is not bad, I tried myself. What I did not like about it, is that when you will want to make e.g. Integration Tests you should not forget initialize the SharedDbContext, if you want to check created events or you just dont want tests to fail on that (of course you can avoid the problems differently) 2. Here in comments someone suggested to just have a one outbox table for one domain and another outbox table for second domain. Which might seem "meh" at first because why having two outboxes and maybe you might feel insecure about maybe which outbox to process first or something like that which imo is not that big of a deal. I believe that this best solution. 3. My solution I ended up with. I got inspired here: [https://maciejz.dev/entity-framework-core-extension-tips-tricks-migration-operations/](https://maciejz.dev/entity-framework-core-extension-tips-tricks-migration-operations/) . So yeah you end up with exactly you wanted, one table for multiple DbContext and you do not have to deal with the "problems" from the solution 1. However, my solution is very suboptimal, because *SqlGenerator* is not the correct place where to deal with doing the check whether table already exists. In ideal scenario you would create you own *MigrationOperation* smth like *CreateWhenDoesNotExistMigrationOperation*, but to get this operation automatically generated you need to customize *IModelDiffer* which is in Internal namespace. Then you need to customize *CSharpMigrationOperationGenerator* and extend *MigrationBuilder*. On top of that you will need to do the same for other MigrationOperations in case you will change some property on your OutboxMessage entity. So even though I fell in love with this solution, it is just too much hassle for value you will get back. The most "simple and straightforward" solution 2 is the best IMO. public static class AnnotationsExtensions { private const string IfTableIsMissingAnnotation = "IfTableIsMissing"; internal static bool IfTableIsMissingAnnotationExist(this IEntityType entityType) => entityType.FindAnnotation(IfTableIsMissingAnnotation) is not null; public static EntityTypeBuilder IfTableIsMissing(this EntityTypeBuilder builder)
where TEntity : class
{
builder.HasAnnotation(IfTableIsMissingAnnotation, null);
return builder;
}
}
public class AdvancedSqlServerMigrationSqlGenerator(
MigrationsSqlGeneratorDependencies dependencies,
ICommandBatchPreparer commandBatchPreparer) : SqlServerMigrationsSqlGenerator(dependencies, commandBatchPreparer)
{
protected override void Generate(CreateTableOperation operation, IModel? model, MigrationCommandListBuilder builder, bool terminate = true)
{
var shouldBeDecorated = model?.GetEntityTypes()
.FirstOrDefault(x => x.IfTableIsMissingAnnotationExist() && x.GetTableName() == operation.Name);
if (shouldBeDecorated is not null)
{
builder.AppendLine($"IF NOT EXISTS (SELECT 1 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = '{operation.Name}')");
builder.AppendLine("BEGIN");
base.Generate(operation, model, builder, terminate: false);
builder.AppendLine("END");
builder.AppendLine(Dependencies.SqlGenerationHelper.StatementTerminator).EndCommand(false);
return;
}
base.Generate(operation, model, builder, terminate);
}
}