Code

Support large files#582

Code

Merged

Ceredron merged 78 commits intomainAltinn/altinn-broker:mainfrom feat/big-file-batchingAltinn/altinn-broker:feat/big-file-batchingCopy head branch name to clipboardNov 14, 2024

+940-284

Commits

Commits on Oct 30, 2024

• Handle upload failure

  Ceredron

  committed
  3e7a270

  Copy full SHA for 3e7a270

• Upload in batch

  Ceredron

  committed
  cf91ca8

  Copy full SHA for cf91ca8

• Use length from header, not stream

  Ceredron

  committed
  f709613

  Copy full SHA for f709613

Commits on Oct 31, 2024

• Try without MD5 hash because of CPU spikes

  Ceredron

  committed
  e1f5695

  Copy full SHA for e1f5695

• Measure effect of increased buffer size

  Ceredron

  committed
  7e9e472

  Copy full SHA for 7e9e472

• Testing out new variant

  Ceredron

  committed
  a3a33af

  Copy full SHA for a3a33af

• Cannot re-use blob md5 hashgen for blocks

  Ceredron

  committed
  e74da5f

  Copy full SHA for e74da5f

• Dont use progresshandler

  Ceredron

  committed
  3529fe0

  Copy full SHA for 3529fe0

• Disable this while testing to get better output

  Ceredron

  committed
  da8c3c6

  Copy full SHA for da8c3c6

• Add readiness probe endpoint we can use instead of default.

  Ceredron

  committed
  c282887

  Copy full SHA for c282887

• Another variant

  Ceredron

  committed
  e013cc5

  Copy full SHA for e013cc5

• No buffering and require content length header

  Ceredron

  committed
  8f2c760

  Copy full SHA for 8f2c760

Commits on Nov 1, 2024

• Test blob MD5

  Ceredron

  committed
  ad8591f

  Copy full SHA for ad8591f

• Use content length from header, not from stream, as stream is no longer buffered

  Ceredron

  committed
  f589ad4

  Copy full SHA for f589ad4

Commits on Nov 2, 2024

• Fixed code with retries, md5 and buffer pooling

  Ceredron

  committed
  5e92f6e

  Copy full SHA for 5e92f6e

• Blockstream must be spooled before retrying after exception

  Ceredron

  committed
  4a1d570

  Copy full SHA for 4a1d570

• Fixed logic for block size, and commit them all at once

  Ceredron

  committed
  e3eb28b

  Copy full SHA for e3eb28b

• Int wraparound bug. Cast to long.

  Ceredron

  committed
  d13e2ed

  Copy full SHA for d13e2ed

• There is a global limit on uncommitted blocks so committing intermediately. Also remove logic for calculating block size for now.

  Ceredron

  committed
  d61b83d

  Copy full SHA for d61b83d

Commits on Nov 3, 2024

• Network stream buffers in 1mb chunks, hence Blocks only became 1mb big. Changed to read into our own buffer until it's full. Will try to reduce # buffers in next iteration.

  Ceredron

  committed
  f179d43

  Copy full SHA for f179d43

• Changed so that only the first commit fails on existing blob

  Ceredron

  committed
  55f8568

  Copy full SHA for 55f8568

Commits on Nov 4, 2024

• Try without retries

  Ceredron

  committed
  fff01a8

  Copy full SHA for fff01a8

Commits on Nov 5, 2024

• Test how fast we can read from stream by nopping

  Ceredron

  committed
  e71413b

  Copy full SHA for e71413b

Commits on Nov 6, 2024

• Read from stream using same network buffer size as uploader

  Ceredron

  committed
  a109781

  Copy full SHA for a109781

• Code clean-up

  Ceredron

  committed
  74acec3

  Copy full SHA for 74acec3

• Stream from blob during download

  Ceredron

  committed
  e30cfc6

  Copy full SHA for e30cfc6

• Set md5 on the blob on final commit

  Ceredron

  committed
  a569f47

  Copy full SHA for a569f47

• No need to set metadata

  Ceredron

  committed
  5c53579

  Copy full SHA for 5c53579

• Fixed MD5 hash

  Ceredron

  committed
  dcddc96

  Copy full SHA for dcddc96

Commits on Nov 7, 2024

• Change to use container client to get blockblobclient instead of creating BlockBlobClient uri directly as it did not work with Azurite.

  Ceredron

  committed
  5b38ca1

  Copy full SHA for 5b38ca1

• Pass length parameter from application code instead of injecting http context straight into integration code

  Ceredron

  committed
  0f12d25

  Copy full SHA for 0f12d25

• Simplified some more

  Ceredron

  committed
  48ce9df

  Copy full SHA for 48ce9df

• Deploy both a virusscan storage account and a non-virusscan storage account for every serviceowner because Microsoft Defender works globally on all blobs in an account.

  Ceredron

  committed
  ec5422b

  Copy full SHA for ec5422b

• No need to be async

  Ceredron

  committed
  1cb58e7

  Copy full SHA for 1cb58e7

Commits on Nov 8, 2024

• Re-wrote API/Application/Persistence code to support different storage provider types (with/without virus scan) and relegated Azurite to non-storageprovider (instead handled in app logic).

  Ceredron

  committed
  9428f0d

  Copy full SHA for 9428f0d

• Fixed data layer

  Ceredron

  committed
  331f75e

  Copy full SHA for 331f75e

• Clean-up

  Ceredron

  committed
  7b4bc17

  Copy full SHA for 7b4bc17

• Fixes from manual testing

  Ceredron

  committed
  fe38930

  Copy full SHA for fe38930

• Removed readines endpoint

  Ceredron

  committed
  69f71aa

  Copy full SHA for 69f71aa

• Remove part of health check that depends on stability of resource manager API.

  Ceredron

  committed
  9b950c6

  Copy full SHA for 9b950c6

• Dead code

  Ceredron

  committed
  4dcad50

  Copy full SHA for 4dcad50

• Do not block threads in health check

  Ceredron

  committed
  3ddb3ab

  Copy full SHA for 3ddb3ab

Commits on Nov 9, 2024

• Multi-threaded upload

  Ceredron

  committed
  cd8dbe5

  Copy full SHA for cd8dbe5

Commits on Nov 11, 2024

• Merge from main

  Ceredron

  committed
  6a11f21

  Copy full SHA for 6a11f21

• Fix migration name

  Ceredron

  committed
  3b5be47

  Copy full SHA for 3b5be47

• Remove Azurite as a storage provider type

  Ceredron

  committed
  11481c7

  Copy full SHA for 11481c7

• Fix constraint name

  Ceredron

  committed
  1a71fd8

  Copy full SHA for 1a71fd8

Commits on Nov 12, 2024

• Fix DI

  Ceredron

  committed
  4940258

  Copy full SHA for 4940258

• Fix parallell upload

  Ceredron

  committed
  b731f2d

  Copy full SHA for b731f2d

• Fix serialization of service owner

  Ceredron

  committed
  ce6c7f3

  Copy full SHA for ce6c7f3

• Fix logic for increasing file size beyond 2GB

  Ceredron

  committed
  e2a4a4c

  Copy full SHA for e2a4a4c

• Remove container creation as handled by IaC

  Ceredron

  committed
  9586771

  Copy full SHA for 9586771

Commits on Nov 13, 2024

• More explicit error handling for the two different file size restriction cases

  Ceredron

  committed
  42c03ed

  Copy full SHA for 42c03ed

• Changed ApplicationSettings to be constants instead. Pushing new code is easier when it is a constant than in IaC. It wasnt setup properly in IaC anyway.

  Ceredron

  committed
  023c013

  Copy full SHA for 023c013

• Fixed manifest file shim configuration

  Ceredron

  committed
  e8e592b

  Copy full SHA for e8e592b

• Add azure storage parameters to options because likely environment-specific

  Ceredron

  committed
  bba33ee

  Copy full SHA for bba33ee

• Added LargeFileTester with instructions

  Ceredron

  committed
  ee9f71c

  Copy full SHA for ee9f71c

• Clean-up

  Ceredron

  committed
  e41cd7e

  Copy full SHA for e41cd7e

• Fix for event bus for now

  Ceredron

  committed
  6cf5dd6

  Copy full SHA for 6cf5dd6

• Fix tests

  Ceredron

  committed
  35c8f0d

  Copy full SHA for 35c8f0d

• Increased to theoretical max

  Ceredron

  committed
  1c52161

  Copy full SHA for 1c52161

• Added ConfigureResource to large file test

  Ceredron

  committed
  cd00adb

  Copy full SHA for cd00adb

• Fix data type

  Ceredron

  committed
  c5ba8c5

  Copy full SHA for c5ba8c5

Commits on Nov 14, 2024

• Added some comments

  Ceredron

  committed
  42362f1

  Copy full SHA for 42362f1

• Initialize list as empty

  Ceredron

  committed
  dd261e5

  Copy full SHA for dd261e5

• Remove defunct code

  Ceredron

  committed
  b527642

  Copy full SHA for b527642

• Default to empty list

  Ceredron

  committed
  7e96772

  Copy full SHA for 7e96772

• Delete blob on failure, not container

  Ceredron

  committed
  50fa891

  Copy full SHA for 50fa891

• Added JsonPropertyName to DisableVirusScan

  Ceredron

  committed
  d0506cc

  Copy full SHA for d0506cc

• Restore code

  Ceredron

  committed
  6d56155

  Copy full SHA for 6d56155

• Ignore offset for random bytes and dispose timer

  Ceredron

  committed
  9e83300

  Copy full SHA for 9e83300

• UTF8 instead of ascii

  Ceredron

  committed
  6503578

  Copy full SHA for 6503578

• Trailing newline

  Ceredron

  committed
  6533991

  Copy full SHA for 6533991

• Inject AzureStorageOptions

  Ceredron

  committed
  4222609

  Copy full SHA for 4222609

• Use options instead of constants

  Ceredron

  committed
  1fb3509

  Copy full SHA for 1fb3509

• Dispose disposables

  Ceredron

  committed
  1aa3d2a

  Copy full SHA for 1aa3d2a

• The timeout applies to single operations. All our operations are now restricted by block size so they are small by nature. No need for long timeout anymore.

  Ceredron

  committed
  44195a4

  Copy full SHA for 44195a4

• Added UseVirusScan to external models

  Ceredron

  committed
  138a242

  Copy full SHA for 138a242