diff --git a/Grace.zip b/Grace.zip new file mode 100644 index 0000000..06eb4aa Binary files /dev/null and b/Grace.zip differ diff --git a/docs/Data types in Grace.md b/docs/Data types in Grace.md new file mode 100644 index 0000000..90ad2e6 --- /dev/null +++ b/docs/Data types in Grace.md @@ -0,0 +1,248 @@ +# Data types in Grace + +Grace uses a fairly simple data structure to keep track of everything. It's more robust than Git's, for sure, but it's as simple as I could make it. + +In this document, first, you'll find an Entity Relationship Diagram (ERD) showing the most relevant types. + +After the diagram, you'll find descriptions of each data type. You can skip directly to the data type you're interested in by clicking the corresponding link below: + +- [Owner and Organization](#owner-and-organization-ie-multitenancy) +- [Repository](#repository) +- [Branch](#branch) +- [DirectoryVersion](#directoryversion) +- [Reference](#reference) +- [FileVersion](#fileversion) + +I'm sure the types will evolve a bit as we move towards a 1.0 release, but the overall structure should be stable now. + +After those descriptions, at the bottom of this document, you'll find a [detailed entity relationship diagram](#detailed-entity-relationship-diagram). This ERD is incomplete, and there are, of course, many other data types in Grace. It's meant to illustrate the most interesting parts, to help you understand the structure of a repository and its contents. Please refer to it as you read the explanations of each type. + +## Entity Relationship Diagram + +The diagram below shows the most important data types in Grace, and how they relate to each other. A [more-detailed ERD](#detailed-entity-relationship-diagram) is available at the bottom of this document. + +```mermaid +erDiagram + Owner ||--|{ Organization : "has 1:N" + Organization ||--|{ Repository : "has 1:N" + Repository ||--|{ Branch : "has 1:N" + Branch ||--|{ Reference : "has 0:N" + Repository ||--|{ DirectoryVersion : "has 1:N" + Reference ||--|| DirectoryVersion : "refers to exactly 1" + DirectoryVersion ||--|{ FileVersion : "has 0:N" +``` + +## Owner and Organization; i.e. Multitenancy + +Grace has a lightweight form of multitenancy built-in. This structure is meant to help large version control hosting platforms to integrate Grace with their existing customer and identity systems. + +I've specifically chosen to do have a two-level Owner / Organization structure based on my experience at GitHub. GitHub started with the construct of an Organization, and in recent years has been adding an "Enterprise" construct above Organizations, to allow large companies to have multiple Organizations managed under one structure. Seeing the importance of that feature set to large companies made it an easy decision to just start with a two-level structure. + +It's not my intention for Grace to replace the identity / organization system for any hoster, and that's why there really isn't much in these data types. They're meant to be "hooks" that a hoster can refer to from their identity systems so they can implement whatever management features they need to safely serve Grace repositories. + +Owner and Organization are the least-used of the data types here. They get created relatively infrequently, they get updated even less frequently, and they get deleted not much at all. + +### What about personal accounts? + +For individual users - like personal user accounts on GitHub that don't belong to any organization - Grace will have one Owner and one Organization that is just for that user, and all user-owned repositories would sit under that Organization. + +There's nothing stopping an individual user from having multiple Organizations (unless the hoster prevents it). There's no performance difference either way. + +## Repository + +Now we get to the version control part. + +Repository is where Grace keeps settings that apply to the entire repository, that apply to each branch by default, and that apply to References and DirectoryVersions in the repository. + +Some examples: + +- RepositoryType - Is the repository public or private? +- SearchVisibility - Should the contents of this repository be visible in search? +- Timings for deleting various entities - + - LogicalDeleteDays - How long should a deleted object be kept before being physically deleted? + - SaveDays - How long should Save References be kept? + - CheckpointDays - How long should Checkpoint References be kept? + - DirectoryVersionCacheDays - How long should the memoized contents of the entire directory tree under a DirectoryVersion be kept? + - DiffCacheDays - How long should the memoized results of a Diff between two DirectoryVersions be kept? +- RecordSaves - Should Auto-save be turned on for this repository? + +In general, once a Repository is created and the settings adjusted to taste, the Repository record will be updated very infrequently. + +## Branch + +Branch is where branches in a repository are defined. It just holds settings that apply to the Branch. + +The most important settings there are: + +- ParentBranchId - Which branch is the parent of this branch? +- \<_Reference_\>Enabled - These control which kinds of References are allowed on the Branch + - PromotionEnabled + - CommitEnabled + - CheckpointEnabled + - SaveEnabled + - TagEnabled + - ExternalEnabled + +I'm sure there will be more settings here as we get to v1.0. + +Branches are created and deleted frequently, of course, but they're updated pretty infrequently. + +That might seem weird if you're used to Git. In Grace, when you do things like `grace checkpoint` or `grace commit` you're not updating the status of a Branch; you're creating a new Reference _in_ that branch. Nothing in the Branch itself changes. + +## DirectoryVersion + +DirectoryVersion holds the data for a specific version of a directory anywhere in a repo. Every time a file in a directory changes, a new DirectoryVersion is created that holds the new state of the directory. If the contents of a subdirectory change, that directory will get a new DirectoryVersion, and so will the next directory up the tree, until we reach the root of the repository. + +In other words, DirectoryVersion is how we capture each unique state in a repository. + +One interesting thing here is that, like the other entities here, Grace uses a Guid for the primary key DirectoryVersionId, and does not use the Sha256Hash as the unique key (even though it always will be unique). My reason for choosing to have an artificial key instead of just using the Sha256Hash is the challenge that Git has had, and is having, migrating to SHA-256, given how deeply embedded SHA-1 is in the naming of objects in Git. It seems best to keep Sha256Hash as a data field, and not as a key, to make it easier to change the hash algorithm in the future. + +Also, DirectoryVersion has the RepositoryId it belongs to, but does not keep a BranchId. This is because a unique version of the Repository, i.e. a DirectoryVersion, can be pointed to from multiple References and from multiple Branches. + +So, DirectoryVersion contains: + +- DirectoryVersionId - This is a Guid that uniquely identifies each DirectoryVersion. +- RepositoryId - not BranchId +- Sha256Hash - Computed over the contents of the directory; the algorithms for computing the Sha256Hash of a [file](https://github.com/ScottArbeit/Grace/blob/337ed395b7f5d033ceb9d178b4fd9442fa383ee5/src/Grace.Shared/Services.Shared.fs#L53) and a [directory](https://github.com/ScottArbeit/Grace/blob/337ed395b7f5d033ceb9d178b4fd9442fa383ee5/src/Grace.Shared/Services.Shared.fs#L92) are in [Services.Shared.fs](https://github.com/ScottArbeit/Grace/blob/main/src/Grace.Shared/Services.Shared.fs). +- RelativePath - no leading '/'; for instance `src/foo/bar.fs` +- Directories - a list of DirectoryVersionId's that refer to the sub-DirectoryVersions. +- Files - a list of FileVersions, one for each not-ignored file in the directory +- Size - int64 + +DirectoryVersions are created and deleted frequently, as References are created and deleted. + +### RootDirectoryVersion + +Because it's such an important construct, in Grace's code you'll see `RootDirectoryVersion` a lot. This is a DirectoryVersion with the path '.', which is the [definition of "root directory"](https://github.com/ScottArbeit/Grace/blob/337ed395b7f5d033ceb9d178b4fd9442fa383ee5/src/Grace.Shared/Constants.Shared.fs#L173-L174) in Grace. Because the RootDirectoryVersion sits at the top of the directory tree, we point to it in a Reference, rather than any sub-DirectoryVersion, as representing a unique version of the repository. + +## Reference + +In Grace, a Reference is how we mark specific RootDirectoryVersions as being interesting in one way or another. + +References have a ReferenceType that indicates what kind it is, so there's no such thing as a Commit entity or a Save entity. They're all just References. + +The interesting parts of a Reference are: + +- ReferenceId - This is a Guid that uniquely identifies each Reference. +- BranchId - The Branch that this Reference is in. A Reference can only be in one Branch. +- DirectoryVersionId - The RootDirectoryVersion that this Reference points to. +- Sha256Hash - The Sha256Hash of the DirectoryVersionId that this Reference points to. Denormalized here for performance reasons. +- ReferenceType - What kind of Reference is this? + - Promotion - This is a Reference that was created by promoting a Commit reference from a child branch to this branch. + - Commit - Commits are candidates for promotion. + - Checkpoint - This is for you to mark a specific version of the repository as being interesting to you. In Git, this is what you'd think of as an intermediate commit as you complete your work. + - Save - These are automatically created by Grace on every save-on-disk, if Auto-Save is turned on. + - Tag - This is a Reference that was created by tagging a Reference. + - External - This is a Reference that was created by an external system, like a CI system. + - Rebase - This is the Reference that gets created when a branch is Rebased on the latest Promotion in its parent branch +- ReferenceType - The attached to the Reference. +- Links - This is a way to link this Reference to another in some relationship. + +References and DirectoryVersions are where the action happens. New References and DirectoryVersions are being created with every save-on-disk (if you have Auto-Save turned on, which you should), and with every checkpoint / commit / promote / tag / external. + +The ratio of new-DirectoryVersions-to-new-References is directly proportional to how deep in the directory tree the updated files are. For every directory level, a new DirectoryVersion will be created. For example, if I update a file called `src/web/js/lib/blah.js` and hit save, that will create one Save Reference, and five new DirectoryVersions - one for the root, and one each for each directory in the path. + +Saves have short lifetimes, and checkpoints (by default) have longer, but finite, lifetimes, and they both get deleted at some point. Any DirectoryVersions that are unique to those references, and any FileVersions in object storage that only appear in those references, get deleted when the Reference is deleted. + +Also, of course, every time a Branch is deleted, all References in that Branch get deleted. And all DirectoryVersions unique to those References get deleted. Etc. + +It's completely normal in Grace for References to be deleted. Happens all the time. + +## FileVersion + +The FileVersion contains the metadata for a file in a DirectoryVersion. It's the metadata for the file, not the file itself. + +The file itself is stored in object storage, and the FileVersion has a BlobUri that points to it. + +The interesting parts of a FileVersion are: + +- RepositoryId - The Repository that this FileVersion is in. +- RelativePath - The path of the file, relative to the Repository root. +- Sha256Hash - The Sha256Hash of the file. +- IsBinary - Is the file binary? +- Size - The size of the file (int64). +- BlobUri - The URI of the file in object storage. + +## Detailed Entity Relationship Diagram + +The diagram below shows the most important data types in Grace, and how they relate to each other. Not every field in each data type is shown - feel free to check out [Types.Shared.fs](https://github.com/ScottArbeit/Grace/blob/main/src/Grace.Shared/Types.Shared.fs) and [Dto.Shared.fs](https://github.com/ScottArbeit/Grace/blob/main/src/Grace.Shared/Dto/Dto.Shared.fs) to see the full data types - but this should give you a good idea of how the data is structured. + +```mermaid +erDiagram + Owner ||--|{ Organization : "has 1:N" + Owner { + OwnerId Guid + OwnerName string + OwnerType OwnerType + SearchVisibility SearchVisibility + } + Organization ||--|{ Repository : "has 1:N" + Organization { + OrganizationId Guid + OrganizationName string + OwnerId Guid + OrganizationType OrganizationType + SearchVisibility SearchVisibility + } + Repository ||--|{ Branch : "has 1:N" + Repository { + RepositoryId Guid + RepositoryName string + OwnerId Guid + OrganizationId Guid + RepositoryType RepositoryType + RepositoryStatus RepositoryStatus + DefaultServerApiVersion string + DefaultBranchName string + LogicalDeleteDays double + SaveDays double + CheckpointDays double + DirectoryVersionCacheDays double + DiffCacheDays double + Description string + RecordSaves bool + } + Branch ||--|{ Reference : "has 1:N" + Branch { + BranchId Guid + BranchName string + OwnerId Guid + OrganizationId Guid + RepositoryId Guid + UserId Guid + PromotionEnabled bool + CommitEnabled bool + CheckpointEnabled bool + SaveEnabled bool + TagEnabled bool + ExternalEnabled bool + AutoRebaseEnabled bool + } + Repository ||--|{ DirectoryVersion : "has 1:N" + Reference ||--|| DirectoryVersion : "refers to exactly 1" + Reference { + ReferenceId Guid + DirectoryVersionId Guid + Sha256Hash string + ReferenceType ReferenceType + ReferenceTest string + Links ReferenceLinkType[] + } + DirectoryVersion { + DirectoryVersionId Guid + RepositoryId Guid + RelativePath string + Sha256Hash string + Directories DirectoryVersionId[] + Files FileVersion[] + } + DirectoryVersion ||--|{ FileVersion : "has 1:N" + FileVersion { + RepositoryId Guid + RelativePath string + Sha256Hash string + IsBinary bool + Size int64 + BlobUri string + } +``` diff --git a/docs/Mermaid diagrams.md b/docs/Mermaid diagrams.md new file mode 100644 index 0000000..daef7ab --- /dev/null +++ b/docs/Mermaid diagrams.md @@ -0,0 +1,54 @@ +# Mermaid diagrams + +## Starting state + +```mermaid +%%{init: { 'logLevel': 'debug', 'theme': 'default', 'gitGraph': {'showBranches': true, 'showCommitLabel': false}} }%% + gitGraph + commit tag: "ce38fa92" + branch Scott + branch Mia + branch Lorenzo + checkout Scott + commit tag: "87923da8: based on ce38fa92" + checkout Mia + commit tag: "7d29abac: based on ce38fa92" + checkout Lorenzo + commit tag: "28a5c67b: based on ce38fa92" + checkout main +``` + +## A promotion on `main` + +```mermaid +%%{init: { 'logLevel': 'debug', 'theme': 'default', 'gitGraph': {'showBranches': true, 'showCommitLabel': false}} }%% + gitGraph + commit tag: "ce38fa92" + branch Scott + branch Mia + branch Lorenzo + checkout Scott + commit tag: "87923da8: based on ce38fa92" + checkout Mia + commit tag: "7d29abac: based on ce38fa92" + checkout Lorenzo + commit tag: "28a5c67b: based on ce38fa92" + checkout main + commit tag: "87923da8" +``` + +## Branching model + +```mermaid +graph TD; + A[master] -->|Merge| B[release]; + B -->|Merge| C[develop]; + C -->|Merge| D[feature branch]; + D -->|Feature Completed| C; + B -->|Release Completed| A; + E[hotfix branch] -->|Fix Applied| A; + E -->|Fix Merged into| C; + + classDef branch fill:#37f,stroke:#666,stroke-width:3px; + class A,B,C,D,E branch; +``` diff --git a/src/Check-CosmosDB-RUs.ps1 b/src/Check-CosmosDB-RUs.ps1 new file mode 100644 index 0000000..08b0991 --- /dev/null +++ b/src/Check-CosmosDB-RUs.ps1 @@ -0,0 +1,30 @@ +# Set variables for the Azure Cosmos DB account and resource group +$resourceGroupName = "gracevcs-development" +$accountName = "gracevcs-development" +$databaseName = "gracevcs-development-db" +$containerName = "grace-development" + +# Function to check the allocated RUs for the specified Cosmos DB account +function Get-CosmosDBRUs { + try { + # Retrieve the current RU settings for the specified container + $ruSettings = az cosmosdb sql container throughput show ` + --resource-group $resourceGroupName ` + --account-name $accountName ` + --database-name $databaseName ` + --name $containerName ` + --query "resource.throughput" ` + --output json + + # Output the current RUs + Write-Host "Current Allocated RUs: $ruSettings" + } catch { + Write-Host "Error fetching RU settings: $_" + } +} + +# Loop to check RUs every minute +while ($true) { + Get-CosmosDBRUs + Start-Sleep -Seconds 60 +} diff --git a/src/CosmosSerializer/CosmosJsonSerializer.csproj b/src/CosmosSerializer/CosmosJsonSerializer.csproj index e09a98c..5a30dc1 100644 --- a/src/CosmosSerializer/CosmosJsonSerializer.csproj +++ b/src/CosmosSerializer/CosmosJsonSerializer.csproj @@ -8,8 +8,11 @@ true - - + + + + + diff --git a/src/Grace.Actors/ActorProxy.Extensions.Actor.fs b/src/Grace.Actors/ActorProxy.Extensions.Actor.fs index f29dcae..b711d68 100644 --- a/src/Grace.Actors/ActorProxy.Extensions.Actor.fs +++ b/src/Grace.Actors/ActorProxy.Extensions.Actor.fs @@ -5,8 +5,10 @@ open Grace.Actors.Extensions.MemoryCache open Grace.Actors.Constants open Grace.Actors.Context open Grace.Actors.Interfaces +open Grace.Shared open Grace.Shared.Constants open Grace.Shared.Types +open Grace.Shared.Utilities open System module ActorProxy = @@ -16,6 +18,7 @@ module ActorProxy = member this.CreateActorProxyWithCorrelationId<'T when 'T :> IActor>(actorId: ActorId, actorType: string, correlationId: CorrelationId) = let actorProxy = actorProxyFactory.CreateActorProxy<'T>(actorId, actorType) memoryCache.CreateCorrelationIdEntry actorId correlationId + //logToConsole $"Created actor proxy: CorrelationId: {correlationId}; ActorType: {actorType}; ActorId: {actorId}." actorProxy module Branch = diff --git a/src/Grace.Actors/Branch.Actor.fs b/src/Grace.Actors/Branch.Actor.fs index 2d6522a..6642e2d 100644 --- a/src/Grace.Actors/Branch.Actor.fs +++ b/src/Grace.Actors/Branch.Actor.fs @@ -21,6 +21,7 @@ open Grace.Shared.Validation.Errors.Branch open Microsoft.Extensions.Logging open System open System.Collections.Generic +open System.Diagnostics open System.Runtime.Serialization open System.Text open System.Threading.Tasks @@ -28,6 +29,7 @@ open NodaTime open System.Text.Json open System.Net.Http.Json open FSharpPlus.Data.MultiMap +open System.Threading module Branch = @@ -257,9 +259,20 @@ module Branch = override this.OnPostActorMethodAsync(context) = let duration_ms = getPaddedDuration_ms actorStartTime - if String.IsNullOrEmpty(currentCommand) then + let threadCount = + match memoryCache.GetThreadCountEntry() with + | Some threadCount -> threadCount + | None -> + let threads = Process.GetCurrentProcess().Threads.Count + memoryCache.CreateThreadCountEntry(threads) + threads + + if + String.IsNullOrEmpty(currentCommand) + && not <| (context.MethodName = "ReceiveReminderAsync") + then log.LogInformation( - "{currentInstant}: Node: {hostName}; Duration: {duration_ms}ms; CorrelationId: {correlationId}; Finished {ActorName}.{MethodName}; RepositoryId: {RepositoryId}; BranchId: {Id}; BranchName: {BranchName}.", + "{currentInstant}: Node: {hostName}; Duration: {duration_ms}ms; CorrelationId: {correlationId}; Finished {ActorName}.{MethodName}; RepositoryId: {RepositoryId}; BranchId: {Id}; BranchName: {BranchName}; ThreadCount: {ThreadCount}.", getCurrentInstantExtended (), getMachineName, duration_ms, @@ -268,11 +281,12 @@ module Branch = context.MethodName, branchDto.RepositoryId, this.Id, - branchDto.BranchName + branchDto.BranchName, + threadCount ) else log.LogInformation( - "{currentInstant}: Node: {hostName}; Duration: {duration_ms}ms; CorrelationId: {correlationId}; Finished {ActorName}.{MethodName}; Command: {Command}; RepositoryId: {RepositoryId}; BranchId: {Id}; BranchName: {BranchName}.", + "{currentInstant}: Node: {hostName}; Duration: {duration_ms}ms; CorrelationId: {correlationId}; Finished {ActorName}.{MethodName}; Command: {Command}; RepositoryId: {RepositoryId}; BranchId: {Id}; BranchName: {BranchName}; ThreadCount: {ThreadCount}.", getCurrentInstantExtended (), getMachineName, duration_ms, @@ -282,7 +296,8 @@ module Branch = currentCommand, branchDto.RepositoryId, this.Id, - branchDto.BranchName + branchDto.BranchName, + threadCount ) logScope.Dispose() @@ -361,6 +376,8 @@ module Branch = let (tuple: PhysicalDeletionReminderState) = (branchDto.RepositoryId, branchDto.BranchId, branchDto.BranchName, branchDto.ParentBranchId, deleteReason, correlationId) + logToConsole $"In Branch.Actor.SchedulePhysicalDeletion: tuple: {tuple}." + this.RegisterReminderAsync(ReminderType.PhysicalDeletion, toByteArray tuple, delay, TimeSpan.FromMilliseconds(-1)) interface IBranchActor with @@ -450,18 +467,13 @@ module Branch = ReferenceType.Rebase [| ReferenceLinkType.BasedOn promotionDto.ReferenceId |] with - | Ok rebaseReferenceId -> - logToConsole $"In BranchActor.Handle.processCommand: rebaseReferenceId: {rebaseReferenceId}." + | Ok rebaseReferenceDto -> + logToConsole $"In BranchActor.Handle.processCommand: rebaseReferenceDto: {rebaseReferenceDto}." | Error error -> logToConsole $"In BranchActor.Handle.processCommand: Error rebasing on referenceId: {basedOn}. promotionDto: {serialize promotionDto}" - use newCacheEntry = - memoryCache.CreateEntry( - $"BrN:{branchName}", - Value = branchId, - AbsoluteExpirationRelativeToNow = MemoryCache.DefaultExpirationTime - ) + memoryCache.CreateBranchNameEntry branchName branchId return Ok(Created(branchId, branchName, parentBranchId, basedOn, repositoryId, branchPermissions)) | BranchCommand.Rebase referenceId -> @@ -484,8 +496,8 @@ module Branch = ReferenceType.Rebase [| ReferenceLinkType.BasedOn promotionDto.ReferenceId |] with - | Ok rebaseReferenceId -> - logToConsole $"In BranchActor.Handle.processCommand: rebaseReferenceId: {rebaseReferenceId}." + | Ok rebaseReferenceDto -> + logToConsole $"In BranchActor.Handle.processCommand: rebaseReferenceDto: {rebaseReferenceDto}." return Ok(Rebased referenceId) | Error error -> logToConsole @@ -537,7 +549,7 @@ module Branch = let! actorReminder = this.SchedulePhysicalDeletion( deleteReason, - TimeSpan.FromDays(repositoryDto.LogicalDeleteDays), + TimeSpan.FromDays(float repositoryDto.LogicalDeleteDays), metadata.CorrelationId ) diff --git a/src/Grace.Actors/Commands.Actor.fs b/src/Grace.Actors/Commands.Actor.fs index 47807bf..4b21d76 100644 --- a/src/Grace.Actors/Commands.Actor.fs +++ b/src/Grace.Actors/Commands.Actor.fs @@ -100,16 +100,16 @@ module Commands = | SetObjectStorageProvider of objectStorageProvider: ObjectStorageProvider | SetStorageAccountName of storageAccountName: StorageAccountName | SetStorageContainerName of storageContainerName: StorageContainerName - | SetVisibility of repositoryVisibility: RepositoryVisibility + | SetRepositoryType of repositoryVisibility: RepositoryType | SetRepositoryStatus of repositoryStatus: RepositoryStatus | SetRecordSaves of recordSaves: bool | SetDefaultServerApiVersion of defaultServerApiVersion: string | SetDefaultBranchName of defaultBranchName: BranchName - | SetLogicalDeleteDays of duration: double - | SetSaveDays of duration: double - | SetCheckpointDays of duration: double - | SetDirectoryVersionCacheDays of duration: double - | SetDiffCacheDays of duration: double + | SetLogicalDeleteDays of duration: single + | SetSaveDays of duration: single + | SetCheckpointDays of duration: single + | SetDirectoryVersionCacheDays of duration: single + | SetDiffCacheDays of duration: single | SetName of repositoryName: RepositoryName | SetDescription of description: string | DeleteLogical of force: bool * DeleteReason: DeleteReason diff --git a/src/Grace.Actors/Constants.Actor.fs b/src/Grace.Actors/Constants.Actor.fs index dd3dd0a..59efe27 100644 --- a/src/Grace.Actors/Constants.Actor.fs +++ b/src/Grace.Actors/Constants.Actor.fs @@ -128,7 +128,7 @@ module Constants = /// The time to wait between logical and physical deletion of an actor's state. /// - /// In Release builds, this is TimeSpan.FromDays(7.0). In Debug builds, it's TimeSpan.FromSeconds(30.0). + /// In Release builds, this is TimeSpan.FromDays(7.0). In Debug builds, it's TimeSpan.FromSeconds(300.0). #if DEBUG let DefaultPhysicalDeletionReminderTime = TimeSpan.FromSeconds(300.0) #else diff --git a/src/Grace.Actors/Diff.Actor.fs b/src/Grace.Actors/Diff.Actor.fs index 5d07385..eecfa24 100644 --- a/src/Grace.Actors/Diff.Actor.fs +++ b/src/Grace.Actors/Diff.Actor.fs @@ -38,6 +38,8 @@ module Diff = let directoryIds = id.GetId().Split("*") (DirectoryVersionId directoryIds[0], DirectoryVersionId directoryIds[1]) + type PhysicalDeletionReminderState = (RepositoryId * DeleteReason * CorrelationId) + type DiffActor(host: ActorHost) = inherit Actor(host) @@ -146,11 +148,10 @@ module Diff = | ObjectStorageProvider.Unknown -> return new MemoryStream() :> Stream } - /// Sets a delete reminder for this actor's state. - member private this.setDeleteReminder() = - let task = this.RegisterReminderAsync(ReminderType.DeleteCachedState, Array.empty, TimeSpan.FromDays(3.0), TimeSpan.FromMilliseconds(-1.0)) - task.Wait() - () + /// Schedules an actor reminder to delete the physical state for this branch. + member private this.SchedulePhysicalDeletion(deleteReason, delay, correlationId) = + let (tuple: PhysicalDeletionReminderState) = (diffDto.RepositoryId, deleteReason, correlationId) + this.RegisterReminderAsync(ReminderType.PhysicalDeletion, toByteArray tuple, delay, TimeSpan.FromMilliseconds(-1)) override this.OnActivateAsync() = let activateStartTime = getCurrentInstant () @@ -254,17 +255,8 @@ module Diff = //logToConsole $"In Actor.Populate(), got differences." - // If there are any differences - there likely are - get the RepositoryDto so we can get download url's. - let! repositoryDto = - task { - if differences.Count > 0 then - let repositoryActorProxy = Repository.CreateActorProxy repositoryId1 correlationId - - let! repositoryDtoFromActor = repositoryActorProxy.Get correlationId - return repositoryDtoFromActor - else - return RepositoryDto.Default - } + let repositoryActorProxy = Repository.CreateActorProxy repositoryId1 correlationId + let! repositoryDto = repositoryActorProxy.Get correlationId /// Gets a Stream for a given RelativePath. let getFileStream (graceIndex: ServerGraceIndex) (relativePath: RelativePath) (repositoryDto: RepositoryDto) = @@ -351,6 +343,7 @@ module Diff = diffDto <- { diffDto with HasDifferences = differences.Count <> 0 + RepositoryId = repositoryDto.RepositoryId DirectoryId1 = directoryId1 Directory1CreatedAt = createdAt1 DirectoryId2 = directoryId2 @@ -359,7 +352,13 @@ module Diff = do! Storage.SaveState stateManager dtoStateName diffDto - this.setDeleteReminder () + let! deletionReminder = + this.SchedulePhysicalDeletion( + ReminderType.DeleteCachedState, + TimeSpan.FromDays(float repositoryDto.DiffCacheDays), + correlationId + ) + return true with ex -> logToConsole $"Exception in DiffActor.Compute(): {createExceptionResponse ex}" diff --git a/src/Grace.Actors/Events.Actor.fs b/src/Grace.Actors/Events.Actor.fs index 7cf2ece..322d987 100644 --- a/src/Grace.Actors/Events.Actor.fs +++ b/src/Grace.Actors/Events.Actor.fs @@ -156,16 +156,16 @@ module Events = | ObjectStorageProviderSet of objectStorageProvider: ObjectStorageProvider | StorageAccountNameSet of storageAccountName: StorageAccountName | StorageContainerNameSet of storageContainerName: StorageContainerName - | RepositoryVisibilitySet of repositoryVisibility: RepositoryVisibility + | RepositoryTypeSet of repositoryVisibility: RepositoryType | RepositoryStatusSet of repositoryStatus: RepositoryStatus | RecordSavesSet of recordSaves: bool | DefaultServerApiVersionSet of defaultServerApiVersion: string | DefaultBranchNameSet of defaultBranchName: BranchName - | LogicalDeleteDaysSet of duration: double - | SaveDaysSet of duration: double - | CheckpointDaysSet of duration: double - | DirectoryVersionCacheDaysSet of duration: double - | DiffCacheDaysSet of duration: double + | LogicalDeleteDaysSet of duration: single + | SaveDaysSet of duration: single + | CheckpointDaysSet of duration: single + | DirectoryVersionCacheDaysSet of duration: single + | DiffCacheDaysSet of duration: single | NameSet of repositoryName: RepositoryName | DescriptionSet of description: string | LogicalDeleted of force: bool * DeleteReason: DeleteReason diff --git a/src/Grace.Actors/Extensions/MemoryCache.Extensions.Actor.fs b/src/Grace.Actors/Extensions/MemoryCache.Extensions.Actor.fs index 3345e89..3a58e25 100644 --- a/src/Grace.Actors/Extensions/MemoryCache.Extensions.Actor.fs +++ b/src/Grace.Actors/Extensions/MemoryCache.Extensions.Actor.fs @@ -184,3 +184,13 @@ module MemoryCache = /// Remove an entry in MemoryCache for a BranchName. member this.RemoveBranchNameEntry(branchName: string) = this.Remove($"{branchNamePrefix}:{branchName}") + + /// Create a new entry in MemoryCache to store the current ThreadCount. + member this.CreateThreadCountEntry(threadCount: int) = + use newCacheEntry = + this.CreateEntry("CurrentThreadCount", Value = threadCount, AbsoluteExpiration = DateTimeOffset.UtcNow.Add(TimeSpan.FromSeconds(30.0))) + + () + + /// Check if we have an entry in MemoryCache for the current ThreadCount. + member this.GetThreadCountEntry() = this.GetFromCache "CurrentThreadCount" diff --git a/src/Grace.Actors/Grace.Actors.fsproj b/src/Grace.Actors/Grace.Actors.fsproj index b07beb2..9090de7 100644 --- a/src/Grace.Actors/Grace.Actors.fsproj +++ b/src/Grace.Actors/Grace.Actors.fsproj @@ -43,17 +43,20 @@ - - - + + + - - + + + - + + + @@ -67,7 +70,7 @@ --> - + diff --git a/src/Grace.Actors/Organization.Actor.fs b/src/Grace.Actors/Organization.Actor.fs index b389682..6f6a6f9 100644 --- a/src/Grace.Actors/Organization.Actor.fs +++ b/src/Grace.Actors/Organization.Actor.fs @@ -303,12 +303,24 @@ module Organization = Task.FromResult(organizationDto) member this.RepositoryExists repositoryName correlationId = - this.correlationId <- correlationId - Task.FromResult(false) + task { + this.correlationId <- correlationId + let actorProxy = RepositoryName.CreateActorProxy organizationDto.OwnerId organizationDto.OrganizationId repositoryName correlationId + + match! actorProxy.GetRepositoryId(correlationId) with + | Some repositoryId -> return true + | None -> return false + } member this.ListRepositories correlationId = - this.correlationId <- correlationId - Task.FromResult(organizationDto.Repositories :> IReadOnlyDictionary) + task { + this.correlationId <- correlationId + let! organizationDtos = Services.getRepositories organizationDto.OrganizationId Int32.MaxValue false + let dict = organizationDtos.ToDictionary((fun repo -> repo.RepositoryId), (fun repo -> repo.RepositoryName)) + + return dict :> IReadOnlyDictionary + } + //Task.FromResult(organizationDto.Repositories :> IReadOnlyDictionary) member this.Handle (command: OrganizationCommand) metadata = let isValid (command: OrganizationCommand) (metadata: EventMetadata) = diff --git a/src/Grace.Actors/Owner.Actor.fs b/src/Grace.Actors/Owner.Actor.fs index 65d5b9e..a8f5833 100644 --- a/src/Grace.Actors/Owner.Actor.fs +++ b/src/Grace.Actors/Owner.Actor.fs @@ -307,16 +307,23 @@ module Owner = ownerDto |> returnTask member this.OrganizationExists organizationName correlationId = - this.correlationId <- correlationId + task { + this.correlationId <- correlationId + let actorProxy = OrganizationName.CreateActorProxy ownerDto.OwnerId organizationName correlationId - ownerDto.Organizations.ContainsValue(OrganizationName organizationName) - |> returnTask + match! actorProxy.GetOrganizationId(correlationId) with + | Some organizationId -> return true + | None -> return false + } member this.ListOrganizations correlationId = - this.correlationId <- correlationId + task { + this.correlationId <- correlationId + let! organizationDtos = Services.getOrganizations ownerDto.OwnerId Int32.MaxValue false + let dict = organizationDtos.ToDictionary((fun org -> org.OrganizationId), (fun org -> org.OrganizationName)) - ownerDto.Organizations :> IReadOnlyDictionary - |> returnTask + return dict :> IReadOnlyDictionary + } member this.Handle command metadata = let isValid command (metadata: EventMetadata) = diff --git a/src/Grace.Actors/Reference.Actor.fs b/src/Grace.Actors/Reference.Actor.fs index 3874008..59d7188 100644 --- a/src/Grace.Actors/Reference.Actor.fs +++ b/src/Grace.Actors/Reference.Actor.fs @@ -257,7 +257,7 @@ module Reference = let! deletionReminder = this.SchedulePhysicalDeletion( $"Deletion for saves of {repositoryDto.SaveDays} days.", - TimeSpan.FromDays(repositoryDto.SaveDays), + TimeSpan.FromDays(float repositoryDto.SaveDays), correlationId ) @@ -272,7 +272,7 @@ module Reference = let! deletionReminder = this.SchedulePhysicalDeletion( $"Deletion for checkpoints of {repositoryDto.CheckpointDays} days.", - TimeSpan.FromDays(repositoryDto.CheckpointDays), + TimeSpan.FromDays(float repositoryDto.CheckpointDays), correlationId ) @@ -357,7 +357,11 @@ module Reference = let! repositoryDto = repositoryActorProxy.Get this.correlationId let! deletionReminder = - this.SchedulePhysicalDeletion(deleteReason, TimeSpan.FromDays(repositoryDto.LogicalDeleteDays), this.correlationId) + this.SchedulePhysicalDeletion( + deleteReason, + TimeSpan.FromDays(float repositoryDto.LogicalDeleteDays), + this.correlationId + ) return LogicalDeleted(force, deleteReason) | DeletePhysical -> diff --git a/src/Grace.Actors/Repository.Actor.fs b/src/Grace.Actors/Repository.Actor.fs index d1773da..899c5b5 100644 --- a/src/Grace.Actors/Repository.Actor.fs +++ b/src/Grace.Actors/Repository.Actor.fs @@ -70,6 +70,8 @@ module Repository = | Some correlationId -> correlationId | None -> String.Empty + logToConsole $"****In Repository.Actor.OnActivateAsync: CorrelationId: {correlationId}; RepositoryId: {this.Id}." + match retrievedDto with | Some retrievedDto -> repositoryDto <- retrievedDto @@ -118,7 +120,10 @@ module Repository = override this.OnPostActorMethodAsync(context) = let duration_ms = getPaddedDuration_ms actorStartTime - if String.IsNullOrEmpty(currentCommand) then + if + String.IsNullOrEmpty(currentCommand) + && not <| (context.MethodName = "ReceiveReminderAsync") + then log.LogInformation( "{currentInstant}: Node: {hostName}; Duration: {duration_ms}ms; CorrelationId: {correlationId}; Finished {ActorName}.{MethodName}; RepositoryId: {Id}.", getCurrentInstantExtended (), @@ -174,7 +179,7 @@ module Repository = | StorageAccountNameSet storageAccountName -> { currentRepositoryDto with StorageAccountName = storageAccountName } | StorageContainerNameSet containerName -> { currentRepositoryDto with StorageContainerName = containerName } | RepositoryStatusSet repositoryStatus -> { currentRepositoryDto with RepositoryStatus = repositoryStatus } - | RepositoryVisibilitySet repositoryVisibility -> { currentRepositoryDto with RepositoryVisibility = repositoryVisibility } + | RepositoryTypeSet repositoryType -> { currentRepositoryDto with RepositoryType = repositoryType } | RecordSavesSet recordSaves -> { currentRepositoryDto with RecordSaves = recordSaves } | DefaultServerApiVersionSet version -> { currentRepositoryDto with DefaultServerApiVersion = version } | DefaultBranchNameSet defaultBranchName -> { currentRepositoryDto with DefaultBranchName = defaultBranchName } @@ -584,7 +589,7 @@ module Repository = | SetStorageAccountName storageAccountName -> return StorageAccountNameSet storageAccountName | SetStorageContainerName containerName -> return StorageContainerNameSet containerName | SetRepositoryStatus repositoryStatus -> return RepositoryStatusSet repositoryStatus - | SetVisibility repositoryVisibility -> return RepositoryVisibilitySet repositoryVisibility + | SetRepositoryType repositoryType -> return RepositoryTypeSet repositoryType | SetRecordSaves recordSaves -> return RecordSavesSet recordSaves | SetDefaultServerApiVersion version -> return DefaultServerApiVersionSet version | SetDefaultBranchName defaultBranchName -> return DefaultBranchNameSet defaultBranchName @@ -607,7 +612,6 @@ module Repository = && branches.Length > 0 && branches.Any(fun branch -> branch.DeletedAt |> Option.isNone) then - //raise (ApplicationException($"{error}")) return LogicalDeleted(force, deleteReason) else // We have --force specified, so delete the branches that aren't already deleted. diff --git a/src/Grace.Actors/Services.Actor.fs b/src/Grace.Actors/Services.Actor.fs index 9baee79..061b2c6 100644 --- a/src/Grace.Actors/Services.Actor.fs +++ b/src/Grace.Actors/Services.Actor.fs @@ -42,7 +42,7 @@ open Microsoft.Extensions.DependencyInjection module Services = type ServerGraceIndex = Dictionary - type OwnerIdRecord = { ownerId: string } + type OwnerIdRecord = { OwnerId: string } type OrganizationIdRecord = { organizationId: string } type RepositoryIdRecord = { repositoryId: string } type BranchIdRecord = { branchId: string } @@ -121,7 +121,7 @@ module Services = let blobContainerClient = BlobContainerClient(azureStorageConnectionString, containerName) // Make sure the container exists before returning the client. - let metadata = Dictionary() :> IDictionary + let metadata = Dictionary(StringComparer.OrdinalIgnoreCase) :> IDictionary metadata[nameof (OwnerId)] <- $"{repositoryDto.OwnerId}" metadata[nameof (OrganizationId)] <- $"{repositoryDto.OrganizationId}" metadata[nameof (RepositoryId)] <- $"{repositoryDto.RepositoryId}" @@ -237,7 +237,7 @@ module Services = QueryDefinition( """SELECT c["value"].OwnerId FROM c - WHERE ENDSWITH(c.id, @stateStorageName) + WHERE c["value"].Class = @stateStorageName AND STRINGEQUALS(c["value"].OwnerName, @ownerName, true)""" ) .WithParameter("@ownerName", ownerName) @@ -248,9 +248,11 @@ module Services = if iterator.HasMoreResults then let! currentResultSet = iterator.ReadNextAsync() - let ownerId = currentResultSet.FirstOrDefault({ ownerId = String.Empty }).ownerId + currentResultSet |> Seq.iter (fun owner -> logToConsole $"Owner: {owner}") + let ownerId = currentResultSet.FirstOrDefault({ OwnerId = String.Empty }).OwnerId if String.IsNullOrEmpty(ownerId) && cacheResultIfNotFound then + logToConsole $"Did not find ownerId using OwnerName {ownerName}. cacheResultIfNotFound: {cacheResultIfNotFound}." // We didn't find the OwnerId, so add this OwnerName to the MemoryCache and indicate that we have already checked. //use newCacheEntry = // memoryCache.CreateEntry( @@ -353,7 +355,7 @@ module Services = // We have already checked and the owner exists. match memoryCache.GetOwnerNameEntry ownerName with | Some ownerGuid -> return Some $"{ownerGuid}" - | None -> // This should never happen, because we just populated the cache in nameExists. + | None -> // This should never happen, because we just populated the cache in ownerNameExists. return None else // The owner name does not exist. @@ -499,8 +501,29 @@ module Services = let repositoryGuid = Guid.Parse(repositoryId) let repositoryActorProxy = Repository.CreateActorProxy repositoryGuid correlationId + log.Value.LogInformation( + "{currentInstant}:****In Services.Actor.repositoryExists, before repositoryActorProxy.Exists: CorrelationId: {correlationId}; RepositoryId: {repositoryId}; repositoryActorProxy.GetHashCode(): {actorProxyHashCode}; threadCount: {threadCount}; threadId: {threadId}.", + getCurrentInstantExtended (), + correlationId, + repositoryId, + repositoryActorProxy.GetHashCode(), + Process.GetCurrentProcess().Threads.Count, + Thread.CurrentThread.ManagedThreadId + ) + let! exists = repositoryActorProxy.Exists correlationId + log.Value.LogInformation( + "{currentInstant}:****In Services.Actor.repositoryExists: CorrelationId: {correlationId}; RepositoryId: {repositoryId}; Exists: {exists}; repositoryActorProxy.GetHashCode(): {actorProxyHashCode}; threadCount: {threadCount}; threadId: {threadId}.", + getCurrentInstantExtended (), + correlationId, + repositoryId, + exists, + repositoryActorProxy.GetHashCode(), + Process.GetCurrentProcess().Threads.Count, + Thread.CurrentThread.ManagedThreadId + ) + if exists then // Add this RepositoryId to the MemoryCache. memoryCache.CreateRepositoryIdEntry repositoryGuid MemoryCache.ExistsValue @@ -831,45 +854,37 @@ module Services = } /// Checks if the specified repository name is unique for the specified organization. - let repositoryNameIsUnique<'T> - (ownerId: string) - (ownerName: string) - (organizationId: string) - (organizationName: string) - (repositoryName: string) - (correlationId: CorrelationId) - = + let repositoryNameIsUnique<'T> (ownerId: string) (organizationId: string) (repositoryName: string) (correlationId: CorrelationId) = task { match actorStateStorageProvider with | Unknown -> return Ok false | AzureCosmosDb -> try - match! resolveOrganizationId ownerId ownerName organizationId organizationName correlationId with - | Some organizationId -> - let queryDefinition = - QueryDefinition( - """SELECT c["value"].RepositoryId FROM c WHERE c["value"].OrganizationId = @organizationId AND c["value"].RepositoryName = @repositoryName AND c["value"].Class = @class""" - ) - .WithParameter("@organizationId", organizationId) - .WithParameter("@repositoryName", repositoryName) - .WithParameter("@class", nameof (RepositoryDto)) - //logToConsole (queryDefinition.QueryText.Replace("@organizationId", $"\"{organizationId}\"").Replace("@repositoryName", $"\"{repositoryName}\"")) - let iterator = cosmosContainer.GetItemQueryIterator(queryDefinition, requestOptions = queryRequestOptions) - - if iterator.HasMoreResults then - let! currentResultSet = iterator.ReadNextAsync() - // If a row is returned, and repositoryId gets a value, then the repository name is not unique. - let repositoryId = currentResultSet.FirstOrDefault({ repositoryId = String.Empty }).repositoryId - - if String.IsNullOrEmpty(repositoryId) then - // The repository name is unique. - return Ok true - else - // The repository name is not unique. - return Ok false + let queryDefinition = + QueryDefinition( + """SELECT c["value"].RepositoryId FROM c WHERE c["value"].OwnerId = @ownerId AND c["value"].OrganizationId = @organizationId AND c["value"].RepositoryName = @repositoryName AND c["value"].Class = @class""" + ) + .WithParameter("@ownerId", ownerId) + .WithParameter("@organizationId", organizationId) + .WithParameter("@repositoryName", repositoryName) + .WithParameter("@class", nameof (RepositoryDto)) + //logToConsole (queryDefinition.QueryText.Replace("@organizationId", $"\"{organizationId}\"").Replace("@repositoryName", $"\"{repositoryName}\"")) + let iterator = cosmosContainer.GetItemQueryIterator(queryDefinition, requestOptions = queryRequestOptions) + + if iterator.HasMoreResults then + let! currentResultSet = iterator.ReadNextAsync() + + // If a row is returned, and repositoryId gets a value, then the repository name is not unique. + let repositoryId = currentResultSet.FirstOrDefault({ repositoryId = String.Empty }).repositoryId + + if String.IsNullOrEmpty(repositoryId) then + // The repository name is unique. + return Ok true else - return Ok true // This else should never be hit. - | None -> return Ok false + // The repository name is not unique. + return Ok false + else + return Ok true // This else should never be hit. with ex -> return Error $"{createExceptionResponse ex}" | MongoDB -> return Ok false @@ -1098,7 +1113,7 @@ module Services = try // (MaxDegreeOfParallelism = 3) runs at 700 RU's, so it fits under the free 1,000 RU limit for CosmosDB, without getting throttled. - let parallelOptions = ParallelOptions(MaxDegreeOfParallelism = 3) + let parallelOptions = ParallelOptions(MaxDegreeOfParallelism = 8) let itemRequestOptions = ItemRequestOptions() diff --git a/src/Grace.Aspire.AppHost/Grace.Aspire.AppHost.csproj b/src/Grace.Aspire.AppHost/Grace.Aspire.AppHost.csproj index 725af16..d62d6c7 100644 --- a/src/Grace.Aspire.AppHost/Grace.Aspire.AppHost.csproj +++ b/src/Grace.Aspire.AppHost/Grace.Aspire.AppHost.csproj @@ -1,23 +1,25 @@ - - Exe - net9.0 - enable - enable - true - + - - - - - - - - - - + + Exe + net9.0 + enable + enable + true + + + + + + + + + + + + diff --git a/src/Grace.Aspire.ServiceDefaults/Grace.Aspire.ServiceDefaults.csproj b/src/Grace.Aspire.ServiceDefaults/Grace.Aspire.ServiceDefaults.csproj index 4e5f84d..dda1998 100644 --- a/src/Grace.Aspire.ServiceDefaults/Grace.Aspire.ServiceDefaults.csproj +++ b/src/Grace.Aspire.ServiceDefaults/Grace.Aspire.ServiceDefaults.csproj @@ -11,8 +11,8 @@ - - + + diff --git a/src/Grace.CLI/Command/Branch.CLI.fs b/src/Grace.CLI/Command/Branch.CLI.fs index b83a24f..9032a69 100644 --- a/src/Grace.CLI/Command/Branch.CLI.fs +++ b/src/Grace.CLI/Command/Branch.CLI.fs @@ -3027,7 +3027,7 @@ module Branch = let! d2 = Directory.GetDirectoryVersionsRecursive(getLatestReferenceDirectoryParameters) let createFileVersionLookupDictionary (directoryVersions: IEnumerable) = - let lookup = Dictionary() + let lookup = Dictionary(StringComparer.OrdinalIgnoreCase) directoryVersions |> Seq.map (fun dv -> dv.ToLocalDirectoryVersion(dv.CreatedAt.ToDateTimeUtc())) @@ -3035,6 +3035,7 @@ module Branch = |> Seq.concat |> Seq.iter (fun file -> lookup.Add(file.RelativePath, file)) + //lookup.GetAlternateLookup() lookup let (directories, errors) = Result.partition [ d1; d2 ] diff --git a/src/Grace.CLI/Command/Connect.CLI.fs b/src/Grace.CLI/Command/Connect.CLI.fs index 4fac23c..9adb296 100644 --- a/src/Grace.CLI/Command/Connect.CLI.fs +++ b/src/Grace.CLI/Command/Connect.CLI.fs @@ -84,7 +84,7 @@ module Connect = if parseResult.HasOption(Options.repositoryId) then match - (Guid.isValidAndNotEmpty commonParameters.RepositoryId InvalidRepositoryId) + (Guid.isValidAndNotEmptyGuid commonParameters.RepositoryId InvalidRepositoryId) .Result with | Ok result -> Result.Ok(parseResult, commonParameters) @@ -107,7 +107,7 @@ module Connect = let mutable ownerId: Guid = Guid.Empty if parseResult.HasOption(Options.ownerId) then - match (Guid.isValidAndNotEmpty commonParameters.OwnerId InvalidOwnerId).Result with + match (Guid.isValidAndNotEmptyGuid commonParameters.OwnerId InvalidOwnerId).Result with | Ok result -> Result.Ok(parseResult, commonParameters) | Error error -> Result.Error error else @@ -126,7 +126,7 @@ module Connect = if parseResult.HasOption(Options.organizationId) then match - (Guid.isValidAndNotEmpty commonParameters.OrganizationId InvalidOrganizationId) + (Guid.isValidAndNotEmptyGuid commonParameters.OrganizationId InvalidOrganizationId) .Result with | Ok result -> Result.Ok(parseResult, commonParameters) diff --git a/src/Grace.CLI/Command/Repository.CLI.fs b/src/Grace.CLI/Command/Repository.CLI.fs index 47afaa9..511e03d 100644 --- a/src/Grace.CLI/Command/Repository.CLI.fs +++ b/src/Grace.CLI/Command/Repository.CLI.fs @@ -98,13 +98,13 @@ module Repository = new Option("--description", IsRequired = false, Description = "The description of the repository.", Arity = ArgumentArity.ExactlyOne) let visibility = - (new Option( + (new Option( "--visibility", IsRequired = true, Description = "The visibility of the repository.", Arity = ArgumentArity.ExactlyOne )) - .FromAmong(listCases ()) + .FromAmong(listCases ()) let status = (new Option("--status", IsRequired = true, Description = "The status of the repository.", Arity = ArgumentArity.ExactlyOne)) @@ -1120,7 +1120,7 @@ module Repository = // Set-SaveDays subcommand type SaveDaysParameters() = inherit CommonParameters() - member val public SaveDays: double = Double.MinValue with get, set + member val public SaveDays: single = Single.MinValue with get, set let private setSaveDaysHandler (parseResult: ParseResult) (parameters: SaveDaysParameters) = task { @@ -1174,7 +1174,7 @@ module Repository = // Set-CheckpointDays subcommand type CheckpointDaysParameters() = inherit CommonParameters() - member val public CheckpointDays: double = Double.MinValue with get, set + member val public CheckpointDays: single = Single.MinValue with get, set let private setCheckpointDaysHandler (parseResult: ParseResult) (parameters: CheckpointDaysParameters) = task { @@ -1227,7 +1227,7 @@ module Repository = type DiffCacheDaysParameters() = inherit CommonParameters() - member val public DiffCacheDays: double = Double.MinValue with get, set + member val public DiffCacheDays: single = Single.MinValue with get, set let private setDiffCacheDaysHandler (parseResult: ParseResult) (parameters: DiffCacheDaysParameters) = task { @@ -1280,7 +1280,7 @@ module Repository = type DirectoryVersionCacheDaysParameters() = inherit CommonParameters() - member val public DirectoryVersionCacheDays: double = Double.MinValue with get, set + member val public DirectoryVersionCacheDays: single = Single.MinValue with get, set let private setDirectoryVersionCacheDaysHandler (parseResult: ParseResult) (parameters: DirectoryVersionCacheDaysParameters) = task { diff --git a/src/Grace.CLI/Grace.CLI.fsproj b/src/Grace.CLI/Grace.CLI.fsproj index 62b61d3..f210c1d 100644 --- a/src/Grace.CLI/Grace.CLI.fsproj +++ b/src/Grace.CLI/Grace.CLI.fsproj @@ -38,24 +38,24 @@ - - - - - + + + + + - - - - + + + + - + @@ -63,6 +63,6 @@ - + \ No newline at end of file diff --git a/src/Grace.Load/Grace.Load.fsproj b/src/Grace.Load/Grace.Load.fsproj index 7294dbe..c3b97aa 100644 --- a/src/Grace.Load/Grace.Load.fsproj +++ b/src/Grace.Load/Grace.Load.fsproj @@ -21,7 +21,7 @@ - + diff --git a/src/Grace.Load/Program.Load.fs b/src/Grace.Load/Program.Load.fs index 514ef34..21f6bb2 100644 --- a/src/Grace.Load/Program.Load.fs +++ b/src/Grace.Load/Program.Load.fs @@ -6,19 +6,18 @@ open Grace.Shared.Parameters open Grace.Shared.Types open Grace.Shared.Utilities open System +open System.Collections.Concurrent open System.Collections.Generic +open System.Diagnostics open System.Linq +open System.Security.Cryptography open System.Threading open System.Threading.Tasks -open System.Collections.Concurrent -open System.Collections.Concurrent -open System.Security.Cryptography -open FSharpPlus.Data.NonEmptySeq module Load = - let numberOfRepositories = 100 - let numberOfBranches = 800 + let numberOfRepositories = 400 + let numberOfBranches = 5000 let numberOfEvents = 100000 let showResult<'T> (r: GraceResult<'T>) = @@ -49,11 +48,11 @@ module Load = let organizationId = Guid.NewGuid() let organizationName = $"Organization{suffixes[0]}" - let! r = Owner.Create(Owner.CreateOwnerParameters(OwnerId = $"{ownerId}", OwnerName = ownerName, CorrelationId = generateCorrelationId ())) + match! Owner.Create(Owner.CreateOwnerParameters(OwnerId = $"{ownerId}", OwnerName = ownerName, CorrelationId = generateCorrelationId ())) with + | Ok result -> logToConsole $"Created owner {ownerId} with OwnerName {ownerName}." + | Error error -> logToConsole $"{error}" - showResult r - - let! r = + match! Organization.Create( Organization.CreateOrganizationParameters( OwnerId = $"{ownerId}", @@ -62,8 +61,33 @@ module Load = CorrelationId = generateCorrelationId () ) ) + with + | Ok result -> logToConsole $"Created organization {organizationId} with OrganizationName {organizationName}." + | Error error -> logToConsole $"{error}" - showResult r + // Warm up the /repository/create path. + let warmupId = Guid.NewGuid().ToString() + + let! warmupRepo = + Repository.Create( + Repository.CreateRepositoryParameters( + OwnerId = $"{ownerId}", + OrganizationId = $"{organizationId}", + RepositoryId = warmupId, + RepositoryName = $"Warmup{suffixes[0]}", + CorrelationId = generateCorrelationId () + ) + ) + + let! deleteWarmupRepo = + Repository.Delete( + Repository.DeleteRepositoryParameters( + OwnerId = $"{ownerId}", + OrganizationId = $"{organizationId}", + RepositoryId = warmupId, + CorrelationId = generateCorrelationId () + ) + ) do! Parallel.ForEachAsync( @@ -72,7 +96,7 @@ module Load = (fun (i: int) (cancellationToken: CancellationToken) -> ValueTask( task { - do! Task.Delay(Random.Shared.Next(25)) + do! Task.Delay(Random.Shared.Next(1000)) let repositoryId = Guid.NewGuid() let repositoryName = $"Repository{suffixes[i]}" @@ -93,7 +117,18 @@ module Load = | Ok r -> logToConsole $"Added repository {i}; repositoryId: {repositoryId}; repositoryName: {repositoryName}." - let! mainBranch = + let! rrrrr = + Repository.SetLogicalDeleteDays( + Repository.SetLogicalDeleteDaysParameters( + OwnerId = $"{ownerId}", + OrganizationId = $"{organizationId}", + RepositoryId = $"{repositoryId}", + LogicalDeleteDays = single (TimeSpan.FromSeconds(45.0).TotalDays), + CorrelationId = generateCorrelationId () + ) + ) + + match! Branch.Get( Branch.GetBranchParameters( OwnerId = $"{ownerId}", @@ -103,12 +138,11 @@ module Load = CorrelationId = generateCorrelationId () ) ) + with + | Ok mainBranch -> + logToConsole $"Adding parentBranchId {i}; mainBranch.ReturnValue.BranchId: {mainBranch.ReturnValue.BranchId}." - match mainBranch with - | Ok branch -> - logToConsole $"Adding parentBranchId {i}; branch.ReturnValue.BranchId: {branch.ReturnValue.BranchId}." - - parentBranchIds.AddOrUpdate(i, branch.ReturnValue.BranchId, (fun _ _ -> branch.ReturnValue.BranchId)) + parentBranchIds.AddOrUpdate(i, mainBranch.ReturnValue.BranchId, (fun _ _ -> mainBranch.ReturnValue.BranchId)) |> ignore | Error error -> logToConsole $"Error getting main: {error}" | Error error -> logToConsole $"Error creating repository: {error}" @@ -125,7 +159,7 @@ module Load = (fun (i: int) (cancellationToken: CancellationToken) -> ValueTask( task { - do! Task.Delay(Random.Shared.Next(50)) + do! Task.Delay(Random.Shared.Next(1000)) let branchId = Guid.NewGuid() let branchName = $"Branch{suffixes[i]}" let repositoryIndex = Random.Shared.Next(repositoryIds.Count) @@ -167,6 +201,10 @@ module Load = let setupTime = getCurrentInstant () logToConsole $"Setup complete. numberOfRepositories: {numberOfRepositories}; numberOfBranches: {numberOfBranches}; ids.Count: {ids.Count}." + logToConsole $"-----------------" + + let mutable chunkStartInstant = getCurrentInstant () + let chunkTransactionsPerSecond = List() do! Parallel.ForEachAsync( @@ -176,8 +214,15 @@ module Load = ValueTask( task { if i % 250 = 0 then + let chunkTPS = float 250 / (getCurrentInstant () - chunkStartInstant).TotalSeconds + chunkTransactionsPerSecond.Add(chunkTPS) + let rollingAverage = chunkTransactionsPerSecond.TakeLast(20).Average() + let totalTPS = float i / (getCurrentInstant () - setupTime).TotalSeconds + logToConsole - $"Processing event {i} of {numberOfEvents}; Transactions/sec: {float i / (getCurrentInstant () - setupTime).TotalSeconds:F3}" + $"Processing event {i} of {numberOfEvents}; Chunk transactions/sec: {chunkTPS:F3}; Rolling average (previous 20): {rollingAverage:F3}; Total transactions/sec: {totalTPS:F3}; Thread count: {Process.GetCurrentProcess().Threads.Count}." + + chunkStartInstant <- getCurrentInstant () let rnd = Random.Shared.Next(ids.Count) let (ownerId, organizationId, repositoryId, branchId) = ids[rnd] @@ -443,7 +488,7 @@ module Load = showResult result let deleteCount = Interlocked.Increment(&deleteCount) - if deleteCount % 100 = 0 then + if deleteCount % 25 = 0 then logToConsole $"Deleted {deleteCount} of {numberOfRepositories} repositories." //do! Task.Delay(Random.Shared.Next(50)) @@ -451,6 +496,8 @@ module Load = )) ) + logToConsole $"Deleted {deleteCount} of {numberOfRepositories} repositories." + let! r = Organization.Delete( Organization.DeleteOrganizationParameters( diff --git a/src/Grace.SDK/Grace.SDK.fsproj b/src/Grace.SDK/Grace.SDK.fsproj index 955eb10..4730837 100644 --- a/src/Grace.SDK/Grace.SDK.fsproj +++ b/src/Grace.SDK/Grace.SDK.fsproj @@ -27,9 +27,9 @@ - + - + @@ -37,7 +37,7 @@ - + diff --git a/src/Grace.SDK/Repository.SDK.fs b/src/Grace.SDK/Repository.SDK.fs index 005aae1..5a2716a 100644 --- a/src/Grace.SDK/Repository.SDK.fs +++ b/src/Grace.SDK/Repository.SDK.fs @@ -1,10 +1,11 @@ -namespace Grace.SDK +namespace Grace.SDK open Grace.SDK.Common open Grace.Shared.Dto.Branch open Grace.Shared.Dto.Reference open Grace.Shared.Dto.Repository open Grace.Shared.Parameters.Repository +open Grace.Shared.Utilities open Grace.Shared.Types open System open System.Collections.Generic @@ -19,6 +20,7 @@ type Repository() = /// /// Values to use when creating the new repository. static member public Create(parameters: CreateRepositoryParameters) = + logToConsole $"Creating repository: RepositoryId: {parameters.RepositoryId}; RepositoryName: {parameters.RepositoryName}." postServer (parameters |> ensureCorrelationIdIsSet, $"repository/{nameof (Repository.Create)}") /// @@ -67,6 +69,10 @@ type Repository() = static member public SetRecordSaves(parameters: RecordSavesParameters) = postServer (parameters |> ensureCorrelationIdIsSet, $"repository/{nameof (Repository.SetRecordSaves)}") + /// Sets the number of days to keep logical deletes in this repository. + static member public SetLogicalDeleteDays(parameters: SetLogicalDeleteDaysParameters) = + postServer (parameters |> ensureCorrelationIdIsSet, $"repository/{nameof (Repository.SetLogicalDeleteDays)}") + /// /// Sets the number of days to keep saves in this repository. /// diff --git a/src/Grace.Server.Tests/General.Server.Tests.fs b/src/Grace.Server.Tests/General.Server.Tests.fs index ab4fd4e..b36be05 100644 --- a/src/Grace.Server.Tests/General.Server.Tests.fs +++ b/src/Grace.Server.Tests/General.Server.Tests.fs @@ -176,9 +176,11 @@ type Setup() = let daprComponentsPath = Path.Combine(homeDirectory, ".dapr", "components-integration") let daprConfigFilePath = Path.Combine(daprComponentsPath, "dapr-config.yaml") + logToTestConsole $"Current process id: {Environment.ProcessId}. Process start time: {Process.GetCurrentProcess().StartTime}." logToTestConsole $"Current directory: {Environment.CurrentDirectory}" let correlationId = generateCorrelationId () + // Delete any existing integration test containers. do! this.DeleteContainers() let daprSchedulerDockerRunArguments = @@ -204,7 +206,7 @@ type Setup() = .WithArguments(daprSchedulerDockerRunArguments) .WithStandardOutputPipe(PipeTarget.ToStringBuilder(sbOutput)) .WithStandardErrorPipe(PipeTarget.ToStringBuilder(sbError)) - .WithValidation(CliWrap.CommandResultValidation.None) + .WithValidation(CommandResultValidation.None) .ExecuteAsync() if daprSchedulerResult.ExitCode = 0 then @@ -273,7 +275,7 @@ type Setup() = .WithArguments(zipkinArguments) .WithStandardOutputPipe(PipeTarget.ToStringBuilder(sbOutput)) .WithStandardErrorPipe(PipeTarget.ToStringBuilder(sbError)) - .WithValidation(CliWrap.CommandResultValidation.None) + .WithValidation(CommandResultValidation.None) .ExecuteAsync() if zipkinResult.ExitCode = 0 then @@ -350,7 +352,7 @@ type Setup() = .WithArguments(daprRuntimeArguments) .WithWorkingDirectory(graceServerPath) .WithEnvironmentVariables(daprEnvironmentVariables) - .WithValidation(CliWrap.CommandResultValidation.None) + .WithValidation(CommandResultValidation.None) .WithStandardOutputPipe(PipeTarget.ToStringBuilder(sbOutput)) .WithStandardErrorPipe(PipeTarget.ToStringBuilder(sbError)) @@ -367,7 +369,7 @@ type Setup() = // Give time for Dapr to warm up. logToTestConsole "Waiting for Dapr to warm up..." - //do! Task.Delay(8000) + while not <| (sbOutput.ToString().Contains("dapr initialized") && (sbOutput.ToString().Contains("Placement order received: unlock"))) do @@ -375,107 +377,151 @@ type Setup() = logToTestConsole "Warm up complete." - // Create the owner we'll use for this test run. try + // Create the owner we'll use for this test run. let ownerParameters = Parameters.Owner.CreateOwnerParameters() ownerParameters.OwnerId <- ownerId - ownerParameters.OwnerName <- $"TestOwner{rnd.Next(1000)}" + ownerParameters.OwnerName <- $"TestOwner{rnd.Next(65535):X4}" ownerParameters.CorrelationId <- correlationId + let! response = Client.PostAsync("/owner/create", createJsonContent ownerParameters) - //logToTestConsole $"Creating owner {ownerParameters.OwnerName} with ID {ownerParameters.OwnerId}. About to call server." + if response.IsSuccessStatusCode then + logToTestConsole $"Owner {ownerParameters.OwnerName} created successfully." + else + let! content = response.Content.ReadAsStringAsync() + Assert.That(content.Length, Is.GreaterThan(0)) + let error = deserialize content + logToTestConsole $"StatusCode: {response.StatusCode}; Content: {error}" - let! response = Client.PostAsync("/owner/create", createJsonContent ownerParameters) + response.EnsureSuccessStatusCode() |> ignore - if not <| response.IsSuccessStatusCode then + // Create the organization we'll use for this test run. + let organizationParameters = Parameters.Organization.CreateOrganizationParameters() + organizationParameters.OwnerId <- ownerId + organizationParameters.OrganizationId <- organizationId + organizationParameters.OrganizationName <- $"TestOrganization{rnd.Next(65535):X4}" + organizationParameters.CorrelationId <- correlationId + let! response = Client.PostAsync("/organization/create", createJsonContent organizationParameters) + + if response.IsSuccessStatusCode then + logToTestConsole $"Organization {organizationParameters.OrganizationName} created successfully." + else let! content = response.Content.ReadAsStringAsync() + Assert.That(content.Length, Is.GreaterThan(0)) let error = deserialize content logToTestConsole $"StatusCode: {response.StatusCode}; Content: {error}" response.EnsureSuccessStatusCode() |> ignore + + // Create the repositories we'll use for this test run. + do! + Parallel.ForEachAsync( + repositoryIds, + Constants.ParallelOptions, + (fun repositoryId ct -> + ValueTask( + task { + let repositoryParameters = Parameters.Repository.CreateRepositoryParameters() + repositoryParameters.OwnerId <- ownerId + repositoryParameters.OrganizationId <- organizationId + repositoryParameters.RepositoryId <- repositoryId + repositoryParameters.RepositoryName <- $"TestRepository{rnd.Next():X8}" + repositoryParameters.CorrelationId <- correlationId + let! response = Client.PostAsync("/repository/create", createJsonContent repositoryParameters) + + if response.IsSuccessStatusCode then + logToTestConsole $"Repository {repositoryParameters.RepositoryName} created successfully." + else + let! content = response.Content.ReadAsStringAsync() + Assert.That(content.Length, Is.GreaterThan(0)) + let error = deserialize content + logToTestConsole $"StatusCode: {response.StatusCode}; Content: {error}" + + response.EnsureSuccessStatusCode() |> ignore + } + )) + ) with ex -> let msg = $"Exception in Setup().{Environment.NewLine}Message: {ex.Message}{Environment.NewLine}Stack trace:{Environment.NewLine}{ex.StackTrace}" logToTestConsole msg - - // Create the organization we'll use for this test run. - let organizationParameters = Parameters.Organization.CreateOrganizationParameters() - organizationParameters.OwnerId <- ownerId - organizationParameters.OrganizationId <- organizationId - organizationParameters.OrganizationName <- $"TestOrganization{rnd.Next(1000)}" - organizationParameters.CorrelationId <- correlationId - let! response = Client.PostAsync("/organization/create", createJsonContent organizationParameters) - let! content = response.Content.ReadAsStringAsync() - //logToTestConsole $"StatusCode: {response.StatusCode}; Content: {content}" - response.EnsureSuccessStatusCode() |> ignore - - // Create the repositories we'll use for this test run. - do! - Parallel.ForEachAsync( - repositoryIds, - Constants.ParallelOptions, - (fun repositoryId ct -> - ValueTask( - task { - let repositoryParameters = Parameters.Repository.CreateRepositoryParameters() - repositoryParameters.OwnerId <- ownerId - repositoryParameters.OrganizationId <- organizationId - repositoryParameters.RepositoryId <- repositoryId - repositoryParameters.RepositoryName <- $"TestRepository{rnd.Next(100000):X4}" - repositoryParameters.CorrelationId <- correlationId - let! response = Client.PostAsync("/repository/create", createJsonContent repositoryParameters) - - let! content = response.Content.ReadAsStringAsync() - response.EnsureSuccessStatusCode() |> ignore - Assert.That(content.Length, Is.GreaterThan(0)) - } - )) - ) + exit -1 } [] member public this.Teardown() = task { - do! - Parallel.ForEachAsync( - repositoryIds, - Constants.ParallelOptions, - (fun repositoryId ct -> - ValueTask( - task { - let repositoryDeleteParameters = Parameters.Repository.DeleteRepositoryParameters() - repositoryDeleteParameters.OwnerId <- ownerId - repositoryDeleteParameters.OrganizationId <- organizationId - repositoryDeleteParameters.RepositoryId <- repositoryId - repositoryDeleteParameters.DeleteReason <- "Deleting test repository" - - let! response = Client.PostAsync("/repository/delete", createJsonContent repositoryDeleteParameters) - - let! content = response.Content.ReadAsStringAsync() - response.EnsureSuccessStatusCode() |> ignore - Assert.That(content.Length, Is.GreaterThan(0)) - } - )) - ) - - let organizationDeleteParameters = Parameters.Organization.DeleteOrganizationParameters() - - organizationDeleteParameters.OwnerId <- ownerId - organizationDeleteParameters.OrganizationId <- organizationId - organizationDeleteParameters.DeleteReason <- "Deleting test organization" - - let! response = Client.PostAsync("/organization/delete", createJsonContent organizationDeleteParameters) - - let ownerDeleteParameters = Parameters.Owner.DeleteOwnerParameters() - ownerDeleteParameters.OwnerId <- ownerId - ownerDeleteParameters.DeleteReason <- "Deleting test owner" - let! response = Client.PostAsync("/owner/delete", createJsonContent ownerDeleteParameters) + let correlationId = generateCorrelationId () - let! content = response.Content.ReadAsStringAsync() - response.EnsureSuccessStatusCode() |> ignore - Assert.That(content.Length, Is.GreaterThan(0)) + try + // Delete the repositories we created for this test run. + do! + Parallel.ForEachAsync( + repositoryIds, + Constants.ParallelOptions, + (fun repositoryId ct -> + ValueTask( + task { + let repositoryDeleteParameters = Parameters.Repository.DeleteRepositoryParameters() + repositoryDeleteParameters.OwnerId <- ownerId + repositoryDeleteParameters.OrganizationId <- organizationId + repositoryDeleteParameters.RepositoryId <- repositoryId + repositoryDeleteParameters.DeleteReason <- "Deleting test repository" + repositoryDeleteParameters.CorrelationId <- correlationId + repositoryDeleteParameters.Force <- true + + let! response = Client.PostAsync("/repository/delete", createJsonContent repositoryDeleteParameters) + + if not <| response.IsSuccessStatusCode then + let! content = response.Content.ReadAsStringAsync() + Assert.That(content.Length, Is.GreaterThan(0)) + let error = deserialize content + logToTestConsole $"StatusCode: {response.StatusCode}; Content: {error}" + + response.EnsureSuccessStatusCode() |> ignore + } + )) + ) + + // Delete the organization we created for this test run. + let organizationDeleteParameters = Parameters.Organization.DeleteOrganizationParameters() + organizationDeleteParameters.OwnerId <- ownerId + organizationDeleteParameters.OrganizationId <- organizationId + organizationDeleteParameters.DeleteReason <- "Deleting test organization" + organizationDeleteParameters.CorrelationId <- correlationId + organizationDeleteParameters.Force <- true + let! response = Client.PostAsync("/organization/delete", createJsonContent organizationDeleteParameters) - do! this.DeleteContainers() + if not <| response.IsSuccessStatusCode then + let! content = response.Content.ReadAsStringAsync() + Assert.That(content.Length, Is.GreaterThan(0)) + let error = deserialize content + logToTestConsole $"StatusCode: {response.StatusCode}; Content: {error}" + + // Delete the owner we created for this test run. + let ownerDeleteParameters = Parameters.Owner.DeleteOwnerParameters() + ownerDeleteParameters.OwnerId <- ownerId + ownerDeleteParameters.DeleteReason <- "Deleting test owner" + ownerDeleteParameters.CorrelationId <- generateCorrelationId () + ownerDeleteParameters.Force <- true + let! response = Client.PostAsync("/owner/delete", createJsonContent ownerDeleteParameters) + + if not <| response.IsSuccessStatusCode then + let! content = response.Content.ReadAsStringAsync() + Assert.That(content.Length, Is.GreaterThan(0)) + let error = deserialize content + logToTestConsole $"StatusCode: {response.StatusCode}; Content: {error}" + + response.EnsureSuccessStatusCode() |> ignore + + // Delete all of the integration test containers. + do! this.DeleteContainers() + with ex -> + let msg = + $"Exception in Teardown().{Environment.NewLine}Message: {ex.Message}{Environment.NewLine}Stack trace:{Environment.NewLine}{ex.StackTrace}" + + logToTestConsole msg } [] diff --git a/src/Grace.Server.Tests/Grace.Server.Tests.fsproj b/src/Grace.Server.Tests/Grace.Server.Tests.fsproj index d9489a2..8d48309 100644 --- a/src/Grace.Server.Tests/Grace.Server.Tests.fsproj +++ b/src/Grace.Server.Tests/Grace.Server.Tests.fsproj @@ -22,10 +22,10 @@ - - - - + + + + all @@ -43,7 +43,7 @@ - + diff --git a/src/Grace.Server.Tests/Repository.Server.Tests.fs b/src/Grace.Server.Tests/Repository.Server.Tests.fs index aee1bf7..8f44249 100644 --- a/src/Grace.Server.Tests/Repository.Server.Tests.fs +++ b/src/Grace.Server.Tests/Repository.Server.Tests.fs @@ -84,7 +84,7 @@ type Repository() = member public this.SetSaveDaysWithValidValues() = task { let parameters = Parameters.Repository.SetSaveDaysParameters() - parameters.SaveDays <- 17.5 + parameters.SaveDays <- 17.5f parameters.OwnerId <- ownerId parameters.OrganizationId <- organizationId parameters.RepositoryId <- repositoryIds[(rnd.Next(0, numberOfRepositories))] @@ -100,7 +100,7 @@ type Repository() = member public this.SetSaveDaysWithInvalidValues() = task { let parameters = Parameters.Repository.SetSaveDaysParameters() - parameters.SaveDays <- -1 + parameters.SaveDays <- -1f parameters.OwnerId <- ownerId parameters.OrganizationId <- organizationId parameters.RepositoryId <- repositoryIds[(rnd.Next(0, numberOfRepositories))] @@ -117,7 +117,7 @@ type Repository() = member public this.SetCheckpointDaysWithValidValues() = task { let parameters = Parameters.Repository.SetCheckpointDaysParameters() - parameters.CheckpointDays <- 17.5 + parameters.CheckpointDays <- 17.5f parameters.OwnerId <- ownerId parameters.OrganizationId <- organizationId parameters.RepositoryId <- repositoryIds[(rnd.Next(0, numberOfRepositories))] @@ -133,7 +133,7 @@ type Repository() = member public this.SetCheckpointDaysWithInvalidValues() = task { let parameters = Parameters.Repository.SetCheckpointDaysParameters() - parameters.CheckpointDays <- -1 + parameters.CheckpointDays <- -1f parameters.OwnerId <- ownerId parameters.OrganizationId <- organizationId parameters.RepositoryId <- repositoryIds[(rnd.Next(0, numberOfRepositories))] diff --git a/src/Grace.Server.Tests/Validations.Server.Tests.fs b/src/Grace.Server.Tests/Validations.Server.Tests.fs index 6804858..baa23d5 100644 --- a/src/Grace.Server.Tests/Validations.Server.Tests.fs +++ b/src/Grace.Server.Tests/Validations.Server.Tests.fs @@ -13,23 +13,23 @@ type Validations() = [] member this.``valid Guid returns Ok``() = - let result = (Guid.isValidAndNotEmpty "6fddb3c1-24c2-4e2e-8f57-98d0838c0c3f" "error").Result + let result = (Guid.isValidAndNotEmptyGuid "6fddb3c1-24c2-4e2e-8f57-98d0838c0c3f" "error").Result Assert.That(result, Is.EqualTo(Common.okResult)) [] member this.``empty string for guid returns Ok``() = - let result = (Guid.isValidAndNotEmpty "" "error").Result + let result = (Guid.isValidAndNotEmptyGuid "" "error").Result Assert.That(result, Is.EqualTo(Common.okResult)) [] member this.``invalid Guid returns Error``() = - let result = (Guid.isValidAndNotEmpty "not a Guid" "error").Result + let result = (Guid.isValidAndNotEmptyGuid "not a Guid" "error").Result Assert.That(result, Is.EqualTo(Common.errorResult)) [] member this.``Guid Empty returns Error``() = - let result = (Guid.isValidAndNotEmpty (Guid.Empty.ToString()) "error").Result + let result = (Guid.isValidAndNotEmptyGuid (Guid.Empty.ToString()) "error").Result Assert.That(result, Is.EqualTo(Common.errorResult)) [] diff --git a/src/Grace.Server/Branch.Server.fs b/src/Grace.Server/Branch.Server.fs index 8ad810e..748b126 100644 --- a/src/Grace.Server/Branch.Server.fs +++ b/src/Grace.Server/Branch.Server.fs @@ -198,27 +198,23 @@ module Branch = let Create: HttpHandler = fun (next: HttpFunc) (context: HttpContext) -> task { + let graceIds = getGraceIds context + let validations (parameters: CreateBranchParameters) = - [| Guid.isValidAndNotEmpty parameters.ParentBranchId InvalidBranchId + [| Guid.isValidAndNotEmptyGuid parameters.ParentBranchId InvalidBranchId String.isValidGraceName parameters.ParentBranchName InvalidBranchName Branch.branchExists - parameters.OwnerId - parameters.OwnerName - parameters.OrganizationId - parameters.OrganizationName - parameters.RepositoryId - parameters.RepositoryName + graceIds.OwnerId + graceIds.OrganizationId + graceIds.RepositoryId parameters.ParentBranchId parameters.ParentBranchName parameters.CorrelationId ParentBranchDoesNotExist Branch.branchNameDoesNotExist parameters.OwnerId - parameters.OwnerName parameters.OrganizationId - parameters.OrganizationName parameters.RepositoryId - parameters.RepositoryName parameters.BranchName parameters.CorrelationId BranchNameAlreadyExists |] @@ -1352,7 +1348,7 @@ module Branch = try let validations (parameters: ListContentsParameters) = [| String.isEmptyOrValidSha256Hash parameters.Sha256Hash InvalidSha256Hash - Guid.isValidAndNotEmpty parameters.ReferenceId InvalidReferenceId |] + Guid.isValidAndNotEmptyGuid parameters.ReferenceId InvalidReferenceId |] let query (context: HttpContext) maxCount (actorProxy: IBranchActor) = task { @@ -1442,7 +1438,7 @@ module Branch = try let validations (parameters: ListContentsParameters) = [| String.isEmptyOrValidSha256Hash parameters.Sha256Hash InvalidSha256Hash - Guid.isValidAndNotEmpty parameters.ReferenceId InvalidReferenceId |] + Guid.isValidAndNotEmptyGuid parameters.ReferenceId InvalidReferenceId |] let query (context: HttpContext) maxCount (actorProxy: IBranchActor) = task { @@ -1535,7 +1531,7 @@ module Branch = try let validations (parameters: GetBranchVersionParameters) = [| String.isEmptyOrValidSha256Hash parameters.Sha256Hash InvalidSha256Hash - Guid.isValidAndNotEmpty parameters.ReferenceId InvalidReferenceId |] + Guid.isValidAndNotEmptyGuid parameters.ReferenceId InvalidReferenceId |] let query (context: HttpContext) maxCount (actorProxy: IBranchActor) = task { diff --git a/src/Grace.Server/DirectoryVersion.Server.fs b/src/Grace.Server/DirectoryVersion.Server.fs index b9a6011..2a0ed78 100644 --- a/src/Grace.Server/DirectoryVersion.Server.fs +++ b/src/Grace.Server/DirectoryVersion.Server.fs @@ -117,9 +117,9 @@ module DirectoryVersion = task { let validations (parameters: CreateParameters) = [| String.isNotEmpty $"{parameters.DirectoryVersion.DirectoryVersionId}" DirectoryVersionError.InvalidDirectoryId - Guid.isValidAndNotEmpty $"{parameters.DirectoryVersion.DirectoryVersionId}" DirectoryVersionError.InvalidDirectoryId + Guid.isValidAndNotEmptyGuid $"{parameters.DirectoryVersion.DirectoryVersionId}" DirectoryVersionError.InvalidDirectoryId String.isNotEmpty $"{parameters.DirectoryVersion.RepositoryId}" DirectoryVersionError.InvalidRepositoryId - Guid.isValidAndNotEmpty $"{parameters.DirectoryVersion.RepositoryId}" DirectoryVersionError.InvalidRepositoryId + Guid.isValidAndNotEmptyGuid $"{parameters.DirectoryVersion.RepositoryId}" DirectoryVersionError.InvalidRepositoryId String.isNotEmpty $"{parameters.DirectoryVersion.RelativePath}" DirectoryVersionError.RelativePathMustNotBeEmpty String.isNotEmpty $"{parameters.DirectoryVersion.Sha256Hash}" DirectoryVersionError.Sha256HashIsRequired String.isValidSha256Hash $"{parameters.DirectoryVersion.Sha256Hash}" DirectoryVersionError.InvalidSha256Hash @@ -143,8 +143,8 @@ module DirectoryVersion = fun (next: HttpFunc) (context: HttpContext) -> task { let validations (parameters: GetParameters) = - [| Guid.isValidAndNotEmpty $"{parameters.RepositoryId}" DirectoryVersionError.InvalidRepositoryId - Guid.isValidAndNotEmpty $"{parameters.DirectoryId}" DirectoryVersionError.InvalidDirectoryId + [| Guid.isValidAndNotEmptyGuid $"{parameters.RepositoryId}" DirectoryVersionError.InvalidRepositoryId + Guid.isValidAndNotEmptyGuid $"{parameters.DirectoryId}" DirectoryVersionError.InvalidDirectoryId Repository.repositoryIdExists $"{parameters.RepositoryId}" parameters.CorrelationId DirectoryVersionError.RepositoryDoesNotExist DirectoryVersion.directoryIdExists (Guid.Parse(parameters.DirectoryId)) @@ -166,8 +166,8 @@ module DirectoryVersion = fun (next: HttpFunc) (context: HttpContext) -> task { let validations (parameters: GetParameters) = - [| Guid.isValidAndNotEmpty $"{parameters.RepositoryId}" DirectoryVersionError.InvalidRepositoryId - Guid.isValidAndNotEmpty $"{parameters.DirectoryId}" DirectoryVersionError.InvalidDirectoryId + [| Guid.isValidAndNotEmptyGuid $"{parameters.RepositoryId}" DirectoryVersionError.InvalidRepositoryId + Guid.isValidAndNotEmptyGuid $"{parameters.DirectoryId}" DirectoryVersionError.InvalidDirectoryId Repository.repositoryIdExists $"{parameters.RepositoryId}" parameters.CorrelationId DirectoryVersionError.RepositoryDoesNotExist DirectoryVersion.directoryIdExists (Guid.Parse(parameters.DirectoryId)) @@ -190,7 +190,7 @@ module DirectoryVersion = fun (next: HttpFunc) (context: HttpContext) -> task { let validations (parameters: GetByDirectoryIdsParameters) = - [| Guid.isValidAndNotEmpty $"{parameters.RepositoryId}" DirectoryVersionError.InvalidRepositoryId + [| Guid.isValidAndNotEmptyGuid $"{parameters.RepositoryId}" DirectoryVersionError.InvalidRepositoryId Repository.repositoryIdExists $"{parameters.RepositoryId}" parameters.CorrelationId DirectoryVersionError.RepositoryDoesNotExist DirectoryVersion.directoryIdsExist parameters.DirectoryIds parameters.CorrelationId DirectoryVersionError.DirectoryDoesNotExist |] @@ -219,8 +219,8 @@ module DirectoryVersion = fun (next: HttpFunc) (context: HttpContext) -> task { let validations (parameters: GetBySha256HashParameters) = - [| Guid.isValidAndNotEmpty $"{parameters.DirectoryId}" DirectoryVersionError.InvalidDirectoryId - Guid.isValidAndNotEmpty $"{parameters.RepositoryId}" DirectoryVersionError.InvalidRepositoryId + [| Guid.isValidAndNotEmptyGuid $"{parameters.DirectoryId}" DirectoryVersionError.InvalidDirectoryId + Guid.isValidAndNotEmptyGuid $"{parameters.RepositoryId}" DirectoryVersionError.InvalidRepositoryId String.isNotEmpty parameters.Sha256Hash DirectoryVersionError.Sha256HashIsRequired Repository.repositoryIdExists $"{parameters.RepositoryId}" parameters.CorrelationId DirectoryVersionError.RepositoryDoesNotExist |] @@ -248,9 +248,9 @@ module DirectoryVersion = for directoryVersion in parameters.DirectoryVersions do let validations = [| String.isNotEmpty $"{directoryVersion.DirectoryVersionId}" DirectoryVersionError.InvalidDirectoryId - Guid.isValidAndNotEmpty $"{directoryVersion.DirectoryVersionId}" DirectoryVersionError.InvalidDirectoryId + Guid.isValidAndNotEmptyGuid $"{directoryVersion.DirectoryVersionId}" DirectoryVersionError.InvalidDirectoryId String.isNotEmpty $"{directoryVersion.RepositoryId}" DirectoryVersionError.InvalidRepositoryId - Guid.isValidAndNotEmpty $"{directoryVersion.RepositoryId}" DirectoryVersionError.InvalidRepositoryId + Guid.isValidAndNotEmptyGuid $"{directoryVersion.RepositoryId}" DirectoryVersionError.InvalidRepositoryId String.isNotEmpty $"{directoryVersion.Sha256Hash}" DirectoryVersionError.Sha256HashIsRequired String.isValidSha256Hash $"{directoryVersion.Sha256Hash}" DirectoryVersionError.InvalidSha256Hash String.isNotEmpty $"{directoryVersion.RelativePath}" DirectoryVersionError.RelativePathMustNotBeEmpty diff --git a/src/Dockerfile b/src/Grace.Server/Dockerfile similarity index 71% rename from src/Dockerfile rename to src/Grace.Server/Dockerfile index 5c16998..d78cdfb 100644 --- a/src/Dockerfile +++ b/src/Grace.Server/Dockerfile @@ -1,8 +1,8 @@ #See https://aka.ms/customizecontainer to learn how to customize your debug container and how Visual Studio uses this Dockerfile to build your images for faster debugging. # Using the SDK image as the base instead of the aspnet image lets us run a shell for debugging. -#FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base -FROM mcr.microsoft.com/dotnet/aspnet:9.0 AS base +#FROM mcr.microsoft.com/dotnet/sdk:9.0 AS base +FROM mcr.microsoft.com/dotnet/sdk:9.0 AS base WORKDIR /app EXPOSE 5000 EXPOSE 5001 @@ -12,14 +12,22 @@ WORKDIR /src COPY ["Grace.Server/Grace.Server.fsproj", "./Grace.Server/"] COPY ["Grace.Actors/Grace.Actors.fsproj", "./Grace.Actors/"] COPY ["Grace.Shared/Grace.Shared.fsproj", "./Grace.Shared/"] +COPY ["Grace.Aspire.ServiceDefaults/Grace.Aspire.ServiceDefaults.csproj", "./Grace.Aspire.ServiceDefaults/"] COPY ["CosmosSerializer/CosmosJsonSerializer.csproj", "./CosmosSerializer/"] RUN dotnet restore "Grace.Server/Grace.Server.fsproj" COPY . . WORKDIR "/src/Grace.Server" -#RUN dotnet build "Grace.Server.fsproj" -c Debug -o /app/build FROM build AS publish -RUN dotnet publish "Grace.Server.fsproj" -c Debug -o /app/publish /p:UseAppHost=false +RUN dotnet build "Grace.Server.fsproj" -c Debug +RUN dotnet publish "Grace.Server.fsproj" -c Debug -o /app/publish --no-build /p:UseAppHost=false + +# Install the dotnet-dump tool globally +RUN dotnet tool install -g dotnet-dump + +# Set the PATH environment variable to include the dotnet tools directory +ENV PATH="$PATH:/root/.dotnet/tools" + # Set environment variables to configure ASP.NET Core to use the certificate #ENV ASPNETCORE_URLS="https://+;http://+" diff --git a/src/Grace.Server/Grace.Server.fsproj b/src/Grace.Server/Grace.Server.fsproj index 590af5b..f4762c1 100644 --- a/src/Grace.Server/Grace.Server.fsproj +++ b/src/Grace.Server/Grace.Server.fsproj @@ -1,4 +1,4 @@ - + net9.0 @@ -7,14 +7,20 @@ 0.1 The server module for Grace Version Control System. f1167a88-7f15-49c3-8ea1-30c2608081c9 - mcr.microsoft.com/dotnet/aspnet:9.0.0-preview.7 + + mcr.microsoft.com/dotnet/sdk:9.0 0.1;latest - scottarbeit/grace-server - - scottarbeit/grace-server + + + + FS0025 false true 1057,3391 @@ -40,6 +46,7 @@ + @@ -63,27 +70,27 @@ - - - + + + - + - - + + - - - - - + + + + + @@ -92,19 +99,14 @@ - - - + + + - + - - - PreserveNewest - - diff --git a/src/Grace.Server/Middleware/Fake.Middleware.fs b/src/Grace.Server/Middleware/Fake.Middleware.fs new file mode 100644 index 0000000..c1984d9 --- /dev/null +++ b/src/Grace.Server/Middleware/Fake.Middleware.fs @@ -0,0 +1,44 @@ +namespace Grace.Server.Middleware + +open Grace.Server +open Grace.Shared +open Grace.Shared.Resources.Text +open Grace.Shared.Utilities +open Microsoft.AspNetCore.Http +open Microsoft.Extensions.Logging +open Microsoft.Extensions.ObjectPool +open System +open System.Text + +/// Checks the incoming request for an X-Correlation-Id header. If there's no CorrelationId header, it generates one and adds it to the response headers. +type FakeMiddleware(next: RequestDelegate) = + + let pooledObjectPolicy = StringBuilderPooledObjectPolicy() + let stringBuilderPool = ObjectPool.Create(pooledObjectPolicy) + let log = ApplicationContext.loggerFactory.CreateLogger(nameof (FakeMiddleware)) + + member this.Invoke(context: HttpContext) = + + // ----------------------------------------------------------------------------------------------------- + // On the way in... +#if DEBUG + let middlewareTraceHeader = context.Request.Headers["X-MiddlewareTraceIn"] + + context.Request.Headers["X-MiddlewareTraceIn"] <- $"{middlewareTraceHeader}{nameof (FakeMiddleware)} --> " +#endif + + let path = context.Request.Path.ToString() + logToConsole $"****In FakeMiddleware; Path: {path}." + + // ----------------------------------------------------------------------------------------------------- + // Pass control to next middleware instance... + let nextTask = next.Invoke(context) + + // ----------------------------------------------------------------------------------------------------- + // On the way out... +#if DEBUG + let middlewareTraceOutHeader = context.Request.Headers["X-MiddlewareTraceOut"] + + context.Request.Headers["X-MiddlewareTraceOut"] <- $"{middlewareTraceOutHeader}{nameof (FakeMiddleware)} --> " +#endif + nextTask diff --git a/src/Grace.Server/Middleware/LogRequestHeaders.Middleware.fs b/src/Grace.Server/Middleware/LogRequestHeaders.Middleware.fs index 8eae5ef..c782270 100644 --- a/src/Grace.Server/Middleware/LogRequestHeaders.Middleware.fs +++ b/src/Grace.Server/Middleware/LogRequestHeaders.Middleware.fs @@ -38,7 +38,7 @@ type LogRequestHeadersMiddleware(next: RequestDelegate) = #endif //let path = context.Request.Path.ToString() - //if path <> "/healthz" then + //if path = "/healthz" then // logToConsole $"In LogRequestHeadersMiddleware.Middleware.fs: Path: {path}." if log.IsEnabled(LogLevel.Debug) then diff --git a/src/Grace.Server/Middleware/ValidateIds.Middleware.fs b/src/Grace.Server/Middleware/ValidateIds.Middleware.fs index 795afea..4b2b38d 100644 --- a/src/Grace.Server/Middleware/ValidateIds.Middleware.fs +++ b/src/Grace.Server/Middleware/ValidateIds.Middleware.fs @@ -190,12 +190,12 @@ type ValidateIdsMiddleware(next: RequestDelegate) = let validations = if path.Equals("/owner/create", StringComparison.InvariantCultureIgnoreCase) then [| Common.String.isNotEmpty ownerId OwnerIdIsRequired - Common.Guid.isValidAndNotEmpty ownerId InvalidOwnerId + Common.Guid.isValidAndNotEmptyGuid ownerId InvalidOwnerId Common.String.isNotEmpty ownerName OwnerNameIsRequired Common.String.isValidGraceName ownerName InvalidOwnerName Common.Input.eitherIdOrNameMustBeProvided ownerId ownerName EitherOwnerIdOrOwnerNameRequired |] else - [| Common.Guid.isValidAndNotEmpty ownerId InvalidOwnerId + [| Common.Guid.isValidAndNotEmptyGuid ownerId InvalidOwnerId Common.String.isValidGraceName ownerName InvalidOwnerName Common.Input.eitherIdOrNameMustBeProvided ownerId ownerName EitherOwnerIdOrOwnerNameRequired |] @@ -229,12 +229,12 @@ type ValidateIdsMiddleware(next: RequestDelegate) = let validations = if path.Equals("/organization/create", StringComparison.InvariantCultureIgnoreCase) then [| Common.String.isNotEmpty organizationId OrganizationIdIsRequired - Common.Guid.isValidAndNotEmpty organizationId InvalidOrganizationId + Common.Guid.isValidAndNotEmptyGuid organizationId InvalidOrganizationId Common.String.isNotEmpty organizationName OrganizationNameIsRequired Common.String.isValidGraceName organizationName InvalidOrganizationName Common.Input.eitherIdOrNameMustBeProvided organizationId organizationName EitherOrganizationIdOrOrganizationNameRequired |] else - [| Common.Guid.isValidAndNotEmpty organizationId InvalidOrganizationId + [| Common.Guid.isValidAndNotEmptyGuid organizationId InvalidOrganizationId Common.String.isValidGraceName organizationName InvalidOrganizationName Common.Input.eitherIdOrNameMustBeProvided organizationId organizationName EitherOrganizationIdOrOrganizationNameRequired |] @@ -269,12 +269,12 @@ type ValidateIdsMiddleware(next: RequestDelegate) = let validations = if path.Equals("/repository/create", StringComparison.InvariantCultureIgnoreCase) then [| Common.String.isNotEmpty repositoryId RepositoryError.RepositoryIdIsRequired - Common.Guid.isValidAndNotEmpty repositoryId RepositoryError.InvalidRepositoryId + Common.Guid.isValidAndNotEmptyGuid repositoryId RepositoryError.InvalidRepositoryId Common.String.isNotEmpty repositoryName RepositoryError.RepositoryNameIsRequired Common.String.isValidGraceName repositoryName RepositoryError.InvalidRepositoryName Common.Input.eitherIdOrNameMustBeProvided repositoryId repositoryName EitherRepositoryIdOrRepositoryNameRequired |] else - [| Common.Guid.isValidAndNotEmpty repositoryId RepositoryError.InvalidRepositoryId + [| Common.Guid.isValidAndNotEmptyGuid repositoryId RepositoryError.InvalidRepositoryId Common.String.isValidGraceName repositoryName RepositoryError.InvalidRepositoryName Common.Input.eitherIdOrNameMustBeProvided repositoryId repositoryName EitherRepositoryIdOrRepositoryNameRequired |] @@ -317,12 +317,12 @@ type ValidateIdsMiddleware(next: RequestDelegate) = let validations = if path.Equals("/branch/create", StringComparison.InvariantCultureIgnoreCase) then [| Common.String.isNotEmpty branchId BranchIdIsRequired - Common.Guid.isValidAndNotEmpty branchId InvalidBranchId + Common.Guid.isValidAndNotEmptyGuid branchId InvalidBranchId Common.String.isNotEmpty branchName BranchNameIsRequired Common.String.isValidGraceName branchName InvalidBranchName Common.Input.eitherIdOrNameMustBeProvided branchId branchName EitherBranchIdOrBranchNameRequired |] else - [| Common.Guid.isValidAndNotEmpty branchId InvalidBranchId + [| Common.Guid.isValidAndNotEmptyGuid branchId InvalidBranchId Common.String.isValidGraceName branchName InvalidBranchName Common.Input.eitherIdOrNameMustBeProvided branchId branchName EitherBranchIdOrBranchNameRequired |] @@ -417,11 +417,18 @@ type ValidateIdsMiddleware(next: RequestDelegate) = path, graceIds.OwnerId ) + // ----------------------------------------------------------------------------------------------------- // Pass control to next middleware instance... + //logToConsole + // $"********About to call next.Invoke(context) in ValidateIds.Middleware.fs. CorrelationId: {correlationId}. Path: {context.Request.Path}; RepositoryId: {graceIds.RepositoryId}." + let nextTask = next.Invoke(context) + //logToConsole + // $"********After call to next.Invoke(context) in ValidateIds.Middleware.fs. CorrelationId: {correlationId}. Path: {context.Request.Path}; RepositoryId: {graceIds.RepositoryId}." + // ----------------------------------------------------------------------------------------------------- // On the way out... diff --git a/src/Grace.Server/Owner.Server.fs b/src/Grace.Server/Owner.Server.fs index 6826360..bb3e8e2 100644 --- a/src/Grace.Server/Owner.Server.fs +++ b/src/Grace.Server/Owner.Server.fs @@ -265,7 +265,7 @@ module Owner = fun (next: HttpFunc) (context: HttpContext) -> task { try - let validations (parameters: ListOrganizationsParameters) = [| Guid.isValidAndNotEmpty parameters.OwnerId InvalidOwnerId |] + let validations (parameters: ListOrganizationsParameters) = [| Guid.isValidAndNotEmptyGuid parameters.OwnerId InvalidOwnerId |] let query (context: HttpContext) (maxCount: int) (actorProxy: IOwnerActor) = task { diff --git a/src/Grace.Server/Repository.Server.fs b/src/Grace.Server/Repository.Server.fs index afe0f8c..0f9b539 100644 --- a/src/Grace.Server/Repository.Server.fs +++ b/src/Grace.Server/Repository.Server.fs @@ -91,16 +91,23 @@ module Repository = return! context |> result400BadRequest graceError } + log.LogInformation( + "{currentInstant}: ****In Repository.Server.processCommand; about to run validations: CorrelationId: {correlationId}; RepositoryId: {repositoryId}.", + getCurrentInstantExtended (), + correlationId, + graceIds.RepositoryId + ) + let validationResults = validations parameters let! validationsPassed = validationResults |> allPass - log.LogDebug( - "{currentInstant}: In Repository.Server.processCommand: RepositoryId: {repositoryId}; validationsPassed: {validationsPassed}; CorrelationId: {correlationId}.", + log.LogInformation( + "{currentInstant}: ****In Repository.Server.processCommand: CorrelationId: {correlationId}; RepositoryId: {repositoryId}; validationsPassed: {validationsPassed}.", getCurrentInstantExtended (), + correlationId, graceIds.RepositoryId, - validationsPassed, - correlationId + validationsPassed ) if validationsPassed then @@ -208,42 +215,31 @@ module Repository = let Create: HttpHandler = fun (next: HttpFunc) (context: HttpContext) -> task { + context.Items.Add("Command", nameof (Create)) + let graceIds = getGraceIds context + //let! parameters = context |> parse - //logToConsole $"parameters.ObjectStorageProvider: {parameters.ObjectStorageProvider}" let validations (parameters: CreateRepositoryParameters) = [| Repository.repositoryIdDoesNotExist parameters.RepositoryId parameters.CorrelationId RepositoryIdAlreadyExists Repository.repositoryNameIsUnique parameters.OwnerId - parameters.OwnerName parameters.OrganizationId - parameters.OrganizationName parameters.RepositoryName parameters.CorrelationId RepositoryNameAlreadyExists |] let command (parameters: CreateRepositoryParameters) = task { - let! ownerId = resolveOwnerId parameters.OwnerId parameters.OwnerName parameters.CorrelationId - - let! organizationId = - resolveOrganizationId - ownerId.Value - parameters.OwnerName - parameters.OrganizationId - parameters.OrganizationName - parameters.CorrelationId - return Create( RepositoryName parameters.RepositoryName, (Guid.Parse(parameters.RepositoryId)), - (Guid.Parse(ownerId.Value)), - (Guid.Parse(organizationId.Value)) + (Guid.Parse(graceIds.OwnerId)), + (Guid.Parse(graceIds.OrganizationId)) ) } |> ValueTask - context.Items.Add("Command", nameof (Create)) return! processCommand context validations command } @@ -256,10 +252,24 @@ module Repository = Repository.repositoryIsNotDeleted context parameters.CorrelationId RepositoryIsDeleted |] let command (parameters: SetRepositoryVisibilityParameters) = - SetVisibility(discriminatedUnionFromString(parameters.Visibility).Value) + SetRepositoryType(discriminatedUnionFromString(parameters.Visibility).Value) |> returnValueTask - context.Items.Add("Command", nameof (SetVisibility)) + context.Items.Add("Command", nameof (SetRepositoryType)) + return! processCommand context validations command + } + + /// Sets the number of days to keep an entity that has been logically deleted. After this time expires, the entity will be physically deleted. + let SetLogicalDeleteDays: HttpHandler = + fun (next: HttpFunc) (context: HttpContext) -> + task { + let validations (parameters: SetLogicalDeleteDaysParameters) = + [| Repository.daysIsValid parameters.LogicalDeleteDays InvalidLogicalDeleteDaysValue + Repository.repositoryIsNotDeleted context parameters.CorrelationId RepositoryIsDeleted |] + + let command (parameters: SetLogicalDeleteDaysParameters) = SetLogicalDeleteDays(parameters.LogicalDeleteDays) |> returnValueTask + + context.Items.Add("Command", nameof (SetLogicalDeleteDays)) return! processCommand context validations command } @@ -392,9 +402,7 @@ module Repository = Repository.repositoryIsNotDeleted context parameters.CorrelationId RepositoryIsDeleted Repository.repositoryNameIsUnique parameters.OwnerId - parameters.OwnerName parameters.OrganizationId - parameters.OrganizationName parameters.NewName parameters.CorrelationId RepositoryNameAlreadyExists |] diff --git a/src/Grace.Server/Services.Server.fs b/src/Grace.Server/Services.Server.fs index a0014f1..89be075 100644 --- a/src/Grace.Server/Services.Server.fs +++ b/src/Grace.Server/Services.Server.fs @@ -87,7 +87,6 @@ module Services = context.SetStatusCode(statusCode) //log.LogDebug("{currentInstant}: In returnResult: StatusCode: {statusCode}; result: {result}", getCurrentInstantExtended(), statusCode, serialize result) - return! context.WriteJsonAsync(result) // .WriteJsonAsync() uses Grace's JsonSerializerOptions. with ex -> let exceptionResponse = Utilities.createExceptionResponse ex diff --git a/src/Grace.Server/Startup.Server.fs b/src/Grace.Server/Startup.Server.fs index 6ddb3ad..cbdb8e7 100644 --- a/src/Grace.Server/Startup.Server.fs +++ b/src/Grace.Server/Startup.Server.fs @@ -59,16 +59,16 @@ module Application = let mustBeLoggedIn = requiresAuthentication notLoggedIn + let fileVersion = + FileVersionInfo + .GetVersionInfo(Assembly.GetExecutingAssembly().Location) + .FileVersion + let endpoints = [ GET [ route "/" (warbler (fun _ -> - let fileVersion = - FileVersionInfo - .GetVersionInfo(Assembly.GetExecutingAssembly().Location) - .FileVersion - htmlString $"

Hello From Grace Server {fileVersion}!


The current server time is: {getCurrentInstantExtended ()}.

")) route @@ -231,6 +231,8 @@ module Application = |> addMetadata typeof route "/setDescription" Repository.SetDescription |> addMetadata typeof + route "/setLogicalDeleteDays" Repository.SetLogicalDeleteDays + |> addMetadata typeof route "/setName" Repository.SetName |> addMetadata typeof route "/setRecordSaves" Repository.SetRecordSaves @@ -496,6 +498,7 @@ module Application = |> ignore app + //.UseMiddleware() .UseW3CLogging() .UseCloudEvents() .UseAuthentication() diff --git a/src/Grace.Server/Validations.Server.fs b/src/Grace.Server/Validations.Server.fs index 554ec91..1c570a6 100644 --- a/src/Grace.Server/Validations.Server.fs +++ b/src/Grace.Server/Validations.Server.fs @@ -350,10 +350,10 @@ module Validations = } |> ValidationResult - let repositoryNameIsUnique<'T> ownerId ownerName organizationId organizationName repositoryName correlationId (error: 'T) = + let repositoryNameIsUnique<'T> ownerId organizationId repositoryName correlationId (error: 'T) = task { if not <| String.IsNullOrEmpty(repositoryName) then - match! repositoryNameIsUnique ownerId ownerName organizationId organizationName repositoryName correlationId with + match! repositoryNameIsUnique ownerId organizationId repositoryName correlationId with | Ok isUnique -> if isUnique then return Ok() else return Error error | Error internalError -> logToConsole internalError @@ -399,39 +399,36 @@ module Validations = |> ValidationResult /// Validates that the branch exists in the database. - let branchExists<'T> ownerId ownerName organizationId organizationName repositoryId repositoryName branchId branchName correlationId (error: 'T) = + let branchExists<'T> ownerId organizationId repositoryId branchId branchName correlationId (error: 'T) = task { let mutable branchGuid = Guid.Empty - match! resolveRepositoryId ownerId ownerName organizationId organizationName repositoryId repositoryName correlationId with - | Some repositoryId -> - match! resolveBranchId repositoryId branchId branchName correlationId with - | Some branchId -> - if Guid.TryParse(branchId, &branchGuid) then - let exists = memoryCache.Get(branchGuid) + match! resolveBranchId repositoryId branchId branchName correlationId with + | Some branchId -> + if Guid.TryParse(branchId, &branchGuid) then + let exists = memoryCache.Get(branchGuid) - match exists with - | MemoryCache.ExistsValue -> return Ok() - | MemoryCache.DoesNotExistValue -> return Error error - | _ -> - let branchActorProxy = Branch.CreateActorProxy branchGuid correlationId - - let! exists = branchActorProxy.Exists correlationId - - if exists then - use newCacheEntry = - memoryCache.CreateEntry( - branchGuid, - Value = MemoryCache.ExistsValue, - AbsoluteExpirationRelativeToNow = MemoryCache.DefaultExpirationTime - ) - - return Ok() - else - return Error error - else - return Error error - | None -> return Error error + match exists with + | MemoryCache.ExistsValue -> return Ok() + | MemoryCache.DoesNotExistValue -> return Error error + | _ -> + let branchActorProxy = Branch.CreateActorProxy branchGuid correlationId + + let! exists = branchActorProxy.Exists correlationId + + if exists then + use newCacheEntry = + memoryCache.CreateEntry( + branchGuid, + Value = MemoryCache.ExistsValue, + AbsoluteExpirationRelativeToNow = MemoryCache.DefaultExpirationTime + ) + + return Ok() + else + return Error error + else + return Error error | None -> return Error error } |> ValidationResult @@ -525,13 +522,10 @@ module Validations = |> ValidationResult /// Validates that the given branchName does not exist in the database. - let branchNameDoesNotExist<'T> ownerId ownerName organizationId organizationName repositoryId repositoryName branchName correlationId (error: 'T) = + let branchNameDoesNotExist<'T> ownerId organizationId repositoryId branchName correlationId (error: 'T) = task { - match! resolveRepositoryId ownerId ownerName organizationId organizationName repositoryId repositoryName correlationId with - | Some repositoryId -> - match! resolveBranchId repositoryId String.Empty branchName correlationId with - | Some branchId -> return Error error - | None -> return Ok() + match! resolveBranchId repositoryId String.Empty branchName correlationId with + | Some branchId -> return Error error | None -> return Ok() } |> ValidationResult diff --git a/src/Grace.Shared/Constants.Shared.fs b/src/Grace.Shared/Constants.Shared.fs index 7cdbb04..e7d6a06 100644 --- a/src/Grace.Shared/Constants.Shared.fs +++ b/src/Grace.Shared/Constants.Shared.fs @@ -105,6 +105,7 @@ module Constants = let GraceObjectCacheFile = "graceObjectCache.json.gz" /// The default branch name for new repositories. + [] let InitialBranchName = "main" /// The configuration version number used by this release of Grace. diff --git a/src/Grace.Shared/Dto/Dto.Shared.fs b/src/Grace.Shared/Dto/Dto.Shared.fs index c170bb6..62d337d 100644 --- a/src/Grace.Shared/Dto/Dto.Shared.fs +++ b/src/Grace.Shared/Dto/Dto.Shared.fs @@ -14,22 +14,24 @@ module Dto = [] type DiffDto = { Class: string - HasDifferences: bool + RepositoryId: RepositoryId DirectoryId1: DirectoryVersionId Directory1CreatedAt: Instant DirectoryId2: DirectoryVersionId Directory2CreatedAt: Instant + HasDifferences: bool Differences: List FileDiffs: List } static member Default = { Class = nameof (DiffDto) - HasDifferences = false + RepositoryId = RepositoryId.Empty DirectoryId1 = DirectoryVersionId.Empty Directory1CreatedAt = Constants.DefaultTimestamp DirectoryId2 = DirectoryVersionId.Empty Directory2CreatedAt = Constants.DefaultTimestamp Differences = List() + HasDifferences = false FileDiffs = List() } static member GetKnownTypes() = GetKnownTypes() @@ -45,7 +47,6 @@ module Dto = OrganizationType: OrganizationType Description: string SearchVisibility: SearchVisibility - Repositories: Dictionary CreatedAt: Instant UpdatedAt: Instant option DeletedAt: Instant option @@ -59,7 +60,6 @@ module Dto = OrganizationType = OrganizationType.Public Description = String.Empty SearchVisibility = Visible - Repositories = new Dictionary() CreatedAt = Constants.DefaultTimestamp UpdatedAt = None DeletedAt = None @@ -77,7 +77,6 @@ module Dto = OwnerType: OwnerType Description: string SearchVisibility: SearchVisibility - Organizations: Dictionary CreatedAt: Instant UpdatedAt: Instant option DeletedAt: Instant option @@ -90,7 +89,6 @@ module Dto = OwnerType = OwnerType.Public Description = String.Empty SearchVisibility = Visible - Organizations = new Dictionary() CreatedAt = Constants.DefaultTimestamp UpdatedAt = None DeletedAt = None @@ -145,16 +143,15 @@ module Dto = ObjectStorageProvider: ObjectStorageProvider StorageAccountName: StorageAccountName StorageContainerName: StorageContainerName - RepositoryVisibility: RepositoryVisibility + RepositoryType: RepositoryType RepositoryStatus: RepositoryStatus - Branches: SortedSet DefaultServerApiVersion: string DefaultBranchName: BranchName - LogicalDeleteDays: double - SaveDays: double - CheckpointDays: double - DirectoryVersionCacheDays: double - DiffCacheDays: double + LogicalDeleteDays: single + SaveDays: single + CheckpointDays: single + DirectoryVersionCacheDays: single + DiffCacheDays: single Description: string RecordSaves: bool CreatedAt: Instant @@ -172,16 +169,15 @@ module Dto = ObjectStorageProvider = ObjectStorageProvider.Unknown StorageAccountName = String.Empty StorageContainerName = "grace-objects" - RepositoryVisibility = RepositoryVisibility.Private + RepositoryType = RepositoryType.Private RepositoryStatus = RepositoryStatus.Active - Branches = SortedSet() DefaultServerApiVersion = "latest" DefaultBranchName = BranchName Constants.InitialBranchName - LogicalDeleteDays = 30.0 - SaveDays = 7.0 - CheckpointDays = 365.0 - DirectoryVersionCacheDays = 1.0 - DiffCacheDays = 1.0 + LogicalDeleteDays = 30.0f + SaveDays = 7.0f + CheckpointDays = 365.0f + DirectoryVersionCacheDays = 1.0f + DiffCacheDays = 1.0f Description = String.Empty RecordSaves = true CreatedAt = Constants.DefaultTimestamp diff --git a/src/Grace.Shared/Grace.Shared.fsproj b/src/Grace.Shared/Grace.Shared.fsproj index e602930..414e7e2 100644 --- a/src/Grace.Shared/Grace.Shared.fsproj +++ b/src/Grace.Shared/Grace.Shared.fsproj @@ -54,22 +54,22 @@ - - + + all runtime; build; native; contentfiles; analyzers; buildtransitive - + - + - + \ No newline at end of file diff --git a/src/Grace.Shared/Parameters/Repository.Parameters.fs b/src/Grace.Shared/Parameters/Repository.Parameters.fs index 0cfe1e1..2538653 100644 --- a/src/Grace.Shared/Parameters/Repository.Parameters.fs +++ b/src/Grace.Shared/Parameters/Repository.Parameters.fs @@ -1,4 +1,4 @@ -namespace Grace.Shared.Parameters +namespace Grace.Shared.Parameters open Grace.Shared.Parameters.Common open Grace.Shared.Types @@ -49,25 +49,30 @@ module Repository = inherit RepositoryParameters() member val public RecordSaves = false with get, set + /// Parameters for the /repository/setLogicalDeleteDays endpoint. + type SetLogicalDeleteDaysParameters() = + inherit RepositoryParameters() + member val public LogicalDeleteDays: single = Single.MinValue with get, set + /// Parameters for the /repository/setSaveDays endpoint. type SetSaveDaysParameters() = inherit RepositoryParameters() - member val public SaveDays: Double = Double.MinValue with get, set + member val public SaveDays: single = Single.MinValue with get, set /// Parameters for the /repository/setCheckpointDays endpoint. type SetCheckpointDaysParameters() = inherit RepositoryParameters() - member val public CheckpointDays: Double = Double.MinValue with get, set + member val public CheckpointDays: single = Single.MinValue with get, set /// Parameters for the /repository/setDirectoryVersionCacheDays endpoint. type SetDirectoryVersionCacheDaysParameters() = inherit RepositoryParameters() - member val public DirectoryVersionCacheDays: Double = Double.MinValue with get, set + member val public DirectoryVersionCacheDays: single = Single.MinValue with get, set /// Parameters for the /repository/setDiffCacheDays endpoint. type SetDiffCacheDaysParameters() = inherit RepositoryParameters() - member val public DiffCacheDays: Double = Double.MinValue with get, set + member val public DiffCacheDays: single = Single.MinValue with get, set /// Parameters for the /repository/setDescription endpoint. type SetRepositoryDescriptionParameters() = diff --git a/src/Grace.Shared/Resources/Text/Languages.Resources.fs b/src/Grace.Shared/Resources/Text/Languages.Resources.fs index 5a0dd76..c6b296f 100644 --- a/src/Grace.Shared/Resources/Text/Languages.Resources.fs +++ b/src/Grace.Shared/Resources/Text/Languages.Resources.fs @@ -65,6 +65,7 @@ module Text = | InvalidDirectoryVersionCacheDaysValue | InvalidDirectoryPath | InvalidDirectoryId + | InvalidLogicalDeleteDaysValue | InvalidMaxCountValue | InvalidObjectStorageProvider | InvalidOrganizationId diff --git a/src/Grace.Shared/Resources/Text/en-US.fs b/src/Grace.Shared/Resources/Text/en-US.fs index 8e14fd5..00b14f0 100644 --- a/src/Grace.Shared/Resources/Text/en-US.fs +++ b/src/Grace.Shared/Resources/Text/en-US.fs @@ -73,6 +73,7 @@ module en_US = | InvalidDirectoryVersionCacheDaysValue -> "The provided value for DirectoryVersionCacheDays is invalid." | InvalidDirectoryPath -> "The provided directory is not a valid directory path." | InvalidDirectoryId -> "The provided DirectoryId is not a valid Guid." + | InvalidLogicalDeleteDaysValue -> "The provided value for LogicalDeleteDays is invalid." | InvalidMaxCountValue -> "The provided value for MaxCount is invalid." | InvalidObjectStorageProvider -> "The provided object storage provider is not valid." | InvalidOrganizationId -> "The provided OrganizationId is not a valid Guid." diff --git a/src/Grace.Shared/Types.Shared.fs b/src/Grace.Shared/Types.Shared.fs index 1298cf0..3a24248 100644 --- a/src/Grace.Shared/Types.Shared.fs +++ b/src/Grace.Shared/Types.Shared.fs @@ -359,11 +359,11 @@ module Types = /// Specifies whether a repository is public or private. [] - type RepositoryVisibility = + type RepositoryType = | Private | Public - static member GetKnownTypes() = GetKnownTypes() + static member GetKnownTypes() = GetKnownTypes() /// Specifies the current operational status of a repository. [] diff --git a/src/Grace.Shared/Validation/Common.Validation.fs b/src/Grace.Shared/Validation/Common.Validation.fs index 40caf9b..d029584 100644 --- a/src/Grace.Shared/Validation/Common.Validation.fs +++ b/src/Grace.Shared/Validation/Common.Validation.fs @@ -12,7 +12,7 @@ module Common = module Guid = /// Validates that a string is a valid Guid, and is not Guid.Empty. - let isValidAndNotEmpty<'T> (s: string) (error: 'T) = + let isValidAndNotEmptyGuid<'T> (s: string) (error: 'T) = if not <| String.IsNullOrEmpty(s) then let mutable guid = Guid.Empty diff --git a/src/Grace.Shared/Validation/Connect.Validation.fs b/src/Grace.Shared/Validation/Connect.Validation.fs index 349a716..2dec0ba 100644 --- a/src/Grace.Shared/Validation/Connect.Validation.fs +++ b/src/Grace.Shared/Validation/Connect.Validation.fs @@ -10,6 +10,6 @@ module Connect = let saveDaysIsAPositiveNumber (saveDays: double) (error: ConnectError) = if saveDays < 0.0 then Error error else Ok() let visibilityIsValid (visibility: string) (error: ConnectError) = - match Utilities.discriminatedUnionFromString (visibility) with + match Utilities.discriminatedUnionFromString (visibility) with | Some visibility -> Ok() | None -> Error error diff --git a/src/Grace.Shared/Validation/Errors.Validation.fs b/src/Grace.Shared/Validation/Errors.Validation.fs index 51c2e62..bd07077 100644 --- a/src/Grace.Shared/Validation/Errors.Validation.fs +++ b/src/Grace.Shared/Validation/Errors.Validation.fs @@ -455,6 +455,7 @@ module Errors = | InvalidDiffCacheDaysValue | InvalidDirectory | InvalidDirectoryVersionCacheDaysValue + | InvalidLogicalDeleteDaysValue | InvalidMaxCountValue | InvalidObjectStorageProvider | InvalidOrganizationId @@ -503,6 +504,7 @@ module Errors = | InvalidDiffCacheDaysValue -> getLocalizedString StringResourceName.InvalidDiffCacheDaysValue | InvalidDirectory -> getLocalizedString StringResourceName.InvalidDirectoryPath | InvalidDirectoryVersionCacheDaysValue -> getLocalizedString StringResourceName.InvalidDirectoryVersionCacheDaysValue + | InvalidLogicalDeleteDaysValue -> getLocalizedString StringResourceName.InvalidLogicalDeleteDaysValue | InvalidMaxCountValue -> getLocalizedString StringResourceName.InvalidMaxCountValue | InvalidObjectStorageProvider -> getLocalizedString StringResourceName.InvalidObjectStorageProvider | InvalidOwnerId -> getLocalizedString StringResourceName.InvalidOwnerId diff --git a/src/Grace.Shared/Validation/Repository.Validation.fs b/src/Grace.Shared/Validation/Repository.Validation.fs index 65d8655..25a8dc3 100644 --- a/src/Grace.Shared/Validation/Repository.Validation.fs +++ b/src/Grace.Shared/Validation/Repository.Validation.fs @@ -1,4 +1,4 @@ -namespace Grace.Shared.Validation +namespace Grace.Shared.Validation open Grace.Shared open Grace.Shared.Types @@ -10,13 +10,13 @@ module Repository = /// Checks that the visibility value provided exists in the RepositoryVisibility type. let visibilityIsValid (visibility: string) (error: RepositoryError) = - match Utilities.discriminatedUnionFromString (visibility) with + match Utilities.discriminatedUnionFromString (visibility) with | Some visibility -> Ok() |> returnValueTask | None -> Error error |> returnValueTask /// Checks that the number of days is between 0.0 and 65536.0. - let daysIsValid (days: double) (error: RepositoryError) = - if days < 0.0 || days >= 65536.0 then + let daysIsValid (days: single) (error: RepositoryError) = + if days < 0.0f || days >= 65536.0f then Error error |> returnValueTask else Ok() |> returnValueTask diff --git a/src/Grace.Shared/Validation/Utilities.Validation.fs b/src/Grace.Shared/Validation/Utilities.Validation.fs index f3420eb..0dcc35d 100644 --- a/src/Grace.Shared/Validation/Utilities.Validation.fs +++ b/src/Grace.Shared/Validation/Utilities.Validation.fs @@ -1,4 +1,4 @@ -namespace Grace.Shared.Validation +namespace Grace.Shared.Validation open FSharp.Control open System.Threading.Tasks @@ -7,13 +7,28 @@ open System module Utilities = /// Returns the first validation that matches the predicate, or None if none match. - let tryFind<'T> (predicate: 'T -> bool) (validations: ValueTask<'T> array) = + let tryFindOld<'T> (predicate: 'T -> bool) (validations: ValueTask<'T> array) = task { match validations |> Seq.tryFindIndex (fun validation -> predicate validation.Result) with | Some index -> return Some(validations[index].Result) | None -> return None } + /// Returns the first validation that matches the predicate, or None if none match. + let tryFind (predicate: 'T -> bool) (validations: ValueTask<'T> array) = + task { + let mutable i = 0 + let mutable first = 0 + + while i < validations.Length && first = 0 do + let! result = validations[i] + if predicate result then first <- i + i <- i + 1 + + // Using .Result here is OK because it would already have been awaited in the while loop above. + if first > 0 then return Some(validations[first].Result) else return None + } + /// Retrieves the first error from a list of validations. let getFirstError (validations: ValueTask> array) = task { diff --git a/src/Grace.sln b/src/Grace.sln index d99117b..fbab279 100644 --- a/src/Grace.sln +++ b/src/Grace.sln @@ -76,7 +76,6 @@ Global {B31CE180-594F-46CF-95FE-4A482EFEC24D}.Release|Any CPU.ActiveCfg = Release|Any CPU {B31CE180-594F-46CF-95FE-4A482EFEC24D}.Release|Any CPU.Build.0 = Release|Any CPU {02A981E8-7BAF-4EAE-B0B1-CD6619F119E9}.Debug|Any CPU.ActiveCfg = Debug|Any CPU - {02A981E8-7BAF-4EAE-B0B1-CD6619F119E9}.Debug|Any CPU.Build.0 = Debug|Any CPU {02A981E8-7BAF-4EAE-B0B1-CD6619F119E9}.Release|Any CPU.ActiveCfg = Release|Any CPU {02A981E8-7BAF-4EAE-B0B1-CD6619F119E9}.Release|Any CPU.Build.0 = Release|Any CPU {6BAB5FC5-3501-4CAB-8507-4EAC4B9D2E5E}.Debug|Any CPU.ActiveCfg = Debug|Any CPU diff --git a/src/Restart-GraceContainer.ps1 b/src/Restart-GraceContainer.ps1 index d996ea8..2290a20 100644 --- a/src/Restart-GraceContainer.ps1 +++ b/src/Restart-GraceContainer.ps1 @@ -1,25 +1,33 @@ $startTime = Get-Date + +# Using Start-Process here to run the deletion of the current deployment asynchronously. Write-Host "Deleting Kubernetes deployment..." Start-Process -FilePath 'C:\Program Files\Docker\Docker\resources\bin\kubectl.exe' -ArgumentList 'delete -f .\kubernetes-deployment.yaml' -#docker build -t "scottarbeit/grace-server:latest" . + $startBulidTime = Get-Date Write-Host "Building the solution..." dotnet build .\Grace.sln -c Debug Write-Host -Write-Host "Creating Docker image..." -dotnet publish .\Grace.Server\Grace.Server.fsproj --no-build -c Debug -p:PublishProfile=DefaultContainer + +Write-Host "Publishing Grace.Server container to local registry..." +dotnet publish .\Grace.Server\Grace.Server.fsproj --no-build -c Debug -t:PublishContainer +#docker build -t "scottarbeit/grace-server:latest,scottarbeit/grace-server:0.1" .\Grace.Server\Dockerfile Write-Host -Write-Host "Publishing Docker image to Docker Hub..." -$startPublishTime = Get-Date + +#Write-Host "Pushing Docker image to Docker Hub..." +#$startPublishTime = Get-Date #docker push scottarbeit/grace-server:latest -Write-Host +#Write-Host +#Write-Host "Docker push time: $([math]::Round(($finishBuildTime - $startPublishTime).TotalSeconds, 2)) seconds" +#Write-Host + $finishBuildTime = Get-Date -Write-Host "Docker push time: $([math]::Round(($finishBuildTime - $startPublishTime).TotalSeconds, 2)) seconds" -Write-Host Write-Host "Build and publish time: $([math]::Round(($finishBuildTime - $startBulidTime).TotalSeconds, 2)) seconds" Write-Host + Write-Host "Restarting Kubernetes deployment..." k apply -f .\kubernetes-deployment.yaml $finishTime = Get-Date Write-Host + Write-Host "Total Time: $([math]::Round(($finishTime - $startTime).TotalSeconds, 2)) seconds" diff --git a/src/kubernetes-deployment-2.yaml b/src/kubernetes-deployment-2.yaml new file mode 100644 index 0000000..1d43166 --- /dev/null +++ b/src/kubernetes-deployment-2.yaml @@ -0,0 +1,423 @@ +apiVersion: v1 +kind: Namespace +metadata: + name: grace-namespace +--- +# PersistentVolumeClaim for Dapr Scheduler's persistent storage +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: dapr-scheduler-pvc + namespace: grace-namespace +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: grace-environment +data: + ASPNETCORE_ENVIRONMENT: Development + ASPNETCORE_HTTP_PORTS: "5000" + ASPNETCORE_HTTPS_PORTS: "5001" + ASPNETCORE_URLS: https://+:5001;http://+:5000 + # ASPNETCORE_Kestrel__Certificates__Default__Path: /https/aspnetapp.pfx + # ASPNETCORE_Kestrel__Certificates__Default__Password: CrypticPassword8=5 + APPLICATIONINSIGHTS_CONNECTION_STRING: "InstrumentationKey=e0955eb4-1817-4a94-bf6e-d48f6ae54a8c;IngestionEndpoint=https://westus2-2.in.applicationinsights.azure.com/" + # Azure_CosmosDB_Connection_String: "AccountEndpoint=https://localhost:8081/;AccountKey=C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==" # This is a well-known default key. + DAPR_APP_ID: "grace-server" + DAPR_APP_PORT: "5000" + DAPR_HTTP_PORT: "3500" + DAPR_GRPC_PORT: "50001" + DAPR_SERVER_URI: http://127.0.0.1 + Logging__LogLevel__Default: "Information" + #Logging__LogLevel__Default: "Debug" + TEMP: /tmp + OTEL_EXPORTER_OTLP_ENDPOINT: "http://otel-collector:4317" + OTEL_RESOURCE_ATTRIBUTES: "service.name=grace-server" +--- +# Deployment for Grace.Server with Dapr sidecar +apiVersion: apps/v1 +kind: Deployment +metadata: + name: grace-server + namespace: grace-namespace + labels: + app: grace-server +spec: + replicas: 1 + selector: + matchLabels: + app: grace-server + template: + metadata: + labels: + app: grace-server + annotations: + # Enable Dapr sidecar injection + dapr.io/enabled: "true" + # Specify Dapr app ID + dapr.io/app-id: "grace-server" + # Expose necessary ports for Dapr + dapr.io/app-port: "5000" + dapr.io/config: "featureconfig" + dapr.io/enable-metrics: "true" + dapr.io/log-level: "debug" + dapr.io/sidecar-cpu-limit: "1.0" + dapr.io/sidecar-memory-limit: "2Gi" + dapr.io/sidecar-cpu-request: "0.25" + dapr.io/sidecar-memory-request: "256Mi" + spec: + volumes: + - name: dapr-config + hostPath: + path: /mnt/c/Users/Scott/.dapr/ + type: Directory + containers: + - name: grace-server + image: scottarbeit/grace-server:latest # Replace with your actual image + resources: + limits: + cpu: "2.0" + memory: "12288Mi" + requests: + cpu: "2.0" + memory: "8192Mi" + envFrom: + - configMapRef: + name: grace-environment + ports: + - containerPort: 5000 + name: http-grace + protocol: TCP + - containerPort: 5001 + name: https-grace + protocol: TCP +--- +# Deployment for Dapr Scheduler with persistent storage +apiVersion: apps/v1 +kind: Deployment +metadata: + name: dapr-scheduler + namespace: grace-namespace + labels: + app: dapr-scheduler +spec: + replicas: 1 + selector: + matchLabels: + app: dapr-scheduler + template: + metadata: + labels: + app: dapr-scheduler + spec: + containers: + - name: dapr-scheduler + image: "daprio/dapr:latest" # Use appropriate Dapr scheduler image + command: ["dapr", "scheduler", "run"] + volumeMounts: + - name: scheduler-storage + mountPath: /data + env: + - name: SCHEDULER_STORAGE_PATH + value: "/data" + volumes: + - name: scheduler-storage + persistentVolumeClaim: + claimName: dapr-scheduler-pvc +--- +# ConfigMap for OpenTelemetry Collector configuration +apiVersion: v1 +kind: ConfigMap +metadata: + name: otel-collector-config + namespace: grace-namespace +data: + otel-collector-config.yaml: | + receivers: + otlp: + protocols: + grpc: + http: + processors: + batch: + exporters: + zipkin: + endpoint: "http://zipkin:9411/api/v2/spans" + prometheus: + endpoint: "0.0.0.0:8889/prometheus" + service: + pipelines: + traces: + receivers: [otlp] + processors: [batch] + exporters: [zipkin] + metrics: + receivers: [otlp] + processors: [batch] + exporters: [prometheus] +--- +# Deployment for OpenTelemetry Collector +apiVersion: apps/v1 +kind: Deployment +metadata: + name: otel-collector + namespace: grace-namespace + labels: + app: otel-collector +spec: + replicas: 1 + selector: + matchLabels: + app: otel-collector + template: + metadata: + labels: + app: otel-collector + spec: + containers: + - name: otel-collector + image: otel/opentelemetry-collector:latest + command: + - "/otelcol" + - "--config=/etc/otel-collector-config.yaml" + ports: + - containerPort: 4317 # OTLP gRPC + - containerPort: 4318 # OTLP HTTP + - containerPort: 8889 # Prometheus metrics + volumeMounts: + - name: config + mountPath: /etc/otel-collector-config.yaml + subPath: otel-collector-config.yaml + volumes: + - name: config + configMap: + name: otel-collector-config +--- +# Deployment for Prometheus +apiVersion: apps/v1 +kind: Deployment +metadata: + name: prometheus + namespace: grace-namespace + labels: + app: prometheus +spec: + replicas: 1 + selector: + matchLabels: + app: prometheus + template: + metadata: + labels: + app: prometheus + spec: + containers: + - name: prometheus + image: prom/prometheus:latest + args: + - "--config.file=/etc/prometheus/prometheus.yml" + ports: + - containerPort: 9090 + volumeMounts: + - name: prometheus-config + mountPath: /etc/prometheus/prometheus.yml + subPath: prometheus.yml + volumes: + - name: prometheus-config + configMap: + name: prometheus-config +--- +# ConfigMap for Prometheus configuration with default scrape settings +apiVersion: v1 +kind: ConfigMap +metadata: + name: prometheus-config + namespace: grace-namespace +data: + prometheus.yml: | + global: + scrape_interval: 15s + scrape_configs: + - job_name: 'grace-server' + static_configs: + - targets: ['grace-server.grace-namespace.svc.cluster.local:80'] + - job_name: 'otel-collector' + static_configs: + - targets: ['otel-collector.grace-namespace.svc.cluster.local:8889'] +--- +# Service for Prometheus +apiVersion: v1 +kind: Service +metadata: + name: prometheus + namespace: grace-namespace + labels: + app: prometheus +spec: + type: ClusterIP + ports: + - port: 9090 + targetPort: 9090 + name: http + selector: + app: prometheus +--- +# Deployment for Zipkin +apiVersion: apps/v1 +kind: Deployment +metadata: + name: zipkin + namespace: grace-namespace + labels: + app: zipkin +spec: + replicas: 1 + selector: + matchLabels: + app: zipkin + template: + metadata: + labels: + app: zipkin + spec: + containers: + - name: zipkin + image: openzipkin/zipkin:latest + ports: + - containerPort: 9411 + env: + - name: STORAGE_TYPE + value: "memory" # Use in-memory storage for development +--- +# Service for Zipkin +apiVersion: v1 +kind: Service +metadata: + name: zipkin + namespace: grace-namespace + labels: + app: zipkin +spec: + type: ClusterIP + ports: + - port: 9411 + targetPort: 9411 + name: http + selector: + app: zipkin +--- +# Service for OpenTelemetry Collector to expose Prometheus metrics +apiVersion: v1 +kind: Service +metadata: + name: otel-collector + namespace: grace-namespace + labels: + app: otel-collector +spec: + type: ClusterIP + ports: + - port: 4317 + targetPort: 4317 + name: otlp-grpc + - port: 4318 + targetPort: 4318 + name: otlp-http + - port: 8889 + targetPort: 8889 + name: prometheus + selector: + app: otel-collector +--- +# Service for Dapr Sidecar (optional if not using Dapr's internal service discovery) +apiVersion: v1 +kind: Service +metadata: + name: grace-server + namespace: grace-namespace + labels: + app: grace-server +spec: + type: ClusterIP + ports: + - port: 80 + targetPort: 80 + name: http + - port: 3500 + targetPort: 3500 + name: dapr + selector: + app: grace-server +--- +# Service for Dapr Scheduler (optional) +apiVersion: v1 +kind: Service +metadata: + name: dapr-scheduler + namespace: grace-namespace + labels: + app: dapr-scheduler +spec: + type: ClusterIP + ports: + - port: 8080 + targetPort: 8080 + name: http + selector: + app: dapr-scheduler +--- +# Ingress (Optional: To expose services externally) +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: grace-ingress + namespace: grace-namespace + annotations: + kubernetes.io/ingress.class: "nginx" # Assuming NGINX Ingress Controller +spec: + rules: + - host: grace.local + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: grace-server + port: + name: http +--- +# Additional ServiceMonitor for Prometheus (if using Prometheus Operator) +# Uncomment if applicable +# apiVersion: monitoring.coreos.com/v1 +# kind: ServiceMonitor +# metadata: +# name: grace-servicemonitor +# namespace: grace-namespace +# labels: +# release: prometheus +# spec: +# selector: +# matchLabels: +# app: grace-server +# endpoints: +# - port: http +# interval: 15s +--- +# Comments: +# - Namespace `grace-namespace` isolates all resources. +# - Dapr sidecar is enabled via annotations in the Grace.Server deployment. +# - PersistentVolumeClaim `dapr-scheduler-pvc` provides storage for Dapr Scheduler. +# - OpenTelemetry Collector is configured to receive OTLP traces and metrics, +# export traces to Zipkin, and expose Prometheus metrics. +# - Prometheus is set up with default scrape configurations targeting Grace.Server and OpenTelemetry Collector. +# - Zipkin uses in-memory storage suitable for development; switch to persistent storage for production. +# - Services expose each component internally within the cluster. +# - Environment variables in Grace.Server configure OpenTelemetry exporters. +# - Ingress is optional and provided for external access using NGINX Ingress Controller. diff --git a/src/kubernetes-deployment.yaml b/src/kubernetes-deployment.yaml index f7bde8a..915789e 100644 --- a/src/kubernetes-deployment.yaml +++ b/src/kubernetes-deployment.yaml @@ -28,7 +28,20 @@ data: TEMP: /tmp --- - +# PersistentVolumeClaim for Dapr Scheduler's persistent storage +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: dapr-scheduler-pvc +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + +--- +# PersistentVolume for Prometheus configuration apiVersion: v1 kind: PersistentVolume metadata: @@ -44,7 +57,7 @@ spec: storageClassName: prometheus-config --- - +# PersistentVolumeClaim for Prometheus configuration apiVersion: v1 kind: PersistentVolumeClaim metadata: @@ -86,11 +99,11 @@ spec: dapr.io/enable-metrics: "true" #dapr.io/env: "GOMEMLIMIT=700MiB" dapr.io/log-level: "debug" - dapr.io/metrics-exporter-type: "prometheus" - dapr.io/metrics-port: "9090" # Ensure this matches the port exposed by the Prometheus exporter in the OpenTelemetry Collector configuration - #dapr.io/placement-host-address: "127.0.0.1:50005" + #dapr.io/metrics-exporter-type: "prometheus" + #dapr.io/metrics-port: "9090" # Ensure this matches the port exposed by the Prometheus exporter in the OpenTelemetry Collector configuration + dapr.io/placement-host-address: "dapr-placement-server.dapr-system.svc.cluster.local:50005" dapr.io/scheduler-host-address: "dapr-scheduler-server.dapr-system.svc.cluster.local:50006" - dapr.io/sidecar-cpu-limit: "1.0" + dapr.io/sidecar-cpu-limit: "4.0" dapr.io/sidecar-memory-limit: "768Mi" dapr.io/sidecar-cpu-request: "0.25" dapr.io/sidecar-memory-request: "256Mi"