Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

High load due to repeated indexing #361

Open
liayn opened this issue Oct 29, 2021 · 3 comments
Open

High load due to repeated indexing #361

liayn opened this issue Oct 29, 2021 · 3 comments

Comments

@liayn
Copy link

liayn commented Oct 29, 2021

Describe the bug
Opening a plain clone of ext:oidc in phpstorm (currently this commit is master xperseguers/t3ext-oidc@c27ec0b) causes massive CPU load due to repeated indexing every few seconds.

The diagnosis shows that the typo3-plugin is causing the indexing with the message
Rebuild requested for index com.cedricziel.idea.typo3.index.core_service_map

This did not happen with other projects to me lately, so this is a nice simple project to reproduce.

(Maybe related to #327 ?)

To Reproduce
Steps to reproduce the behavior:

  1. Clone https://github.com/xperseguers/t3ext-oidc
  2. Open as project
  3. Simply wait and watch the indexing happening

Expected behavior
No indexing, when nothing has changed.

Versions

PhpStorm 2021.2.3
Build #PS-212.5457.49, built on October 13, 2021
Runtime version: 11.0.12+7-b1504.40 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
Windows 10 10.0
GC: ParNew, ConcurrentMarkSweep
Memory: 5700M
Cores: 12
Registry: run.processes.with.pty=TRUE, ide.tooltip.initialDelay=720, ide.images.show.chessboard=true
Non-Bundled Plugins: String Manipulation (8.20.203.000.1), com.urswolfer.intellij.plugin.gerrit (1.2.5-203), YAML/Ansible support (0.11.2), de.sgalinski.fluid.plugin.id (2.5.1), de.sgalinski.typoscript.enterprise.plugin.id (2.5.1), com.cedricziel.idea.typo3 (212.7.2)
@boehmmatthias
Copy link

Same issue - different project.

PhpStorm 2021.3
Build #PS-213.5744.279
Runtime version: 11.0.13+7-b1751.19 amd64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
OS: Ubuntu 20.04.3 LTS

Logs:

2021-12-22 09:03:06,712 [ 235700]   INFO - indexing.UnindexedFilesUpdater - Cancelled indexing of sitesetup 
2021-12-22 09:03:06,712 [ 235700]   INFO - ij.psi.search.LogFileTypeIndex - File type index snapshot dropped 
2021-12-22 09:03:06,712 [ 235700]   INFO - ij.psi.search.LogFileTypeIndex - Loading file type index snapshot 
2021-12-22 09:03:06,713 [ 235701]   INFO - indexing.UnindexedFilesUpdater - Started indexing of sitesetup. Reason: Rebuild requested for index com.cedricziel.idea.typo3.index.core_service_map 
2021-12-22 09:03:06,713 [ 235701]   INFO - indexing.UnindexedFilesUpdater - Performing delayed pushing properties tasks for sitesetup took 0ms; general responsiveness: ok; EDT responsiveness: ok 
2021-12-22 09:03:06,714 [ 235702]   INFO - indexing.UnindexedFilesUpdater - Scanning of sitesetup uses 11 scanning threads 
2021-12-22 09:03:07,422 [ 236410]   INFO - indexing.UnindexedFilesUpdater - Scanning completed for sitesetup. Number of scanned files: 15409; Number of files for indexing: 834 took 709ms; general responsiveness: ok; EDT responsiveness: ok 
2021-12-22 09:03:07,428 [ 236416]   INFO - indexing.UnindexedFilesUpdater - Use 4 indexing threads for indexing of sitesetup 
2021-12-22 09:03:07,433 [ 236421]   INFO - l.indexing.impl.MapReduceIndex - An exception during updateWithMap(). Index com.cedricziel.idea.typo3.index.core_service_map will be rebuilt. 
com.intellij.util.indexing.StorageException: java.lang.IllegalArgumentException: Argument for @NotNull parameter 'value' of com/intellij/util/io/EnumeratorStringDescriptor.save must not be null
	at com.intellij.util.indexing.impl.MapReduceIndex.updateWithMap(MapReduceIndex.java:355)
	at com.intellij.util.indexing.impl.storage.VfsAwareMapReduceIndex.updateWithMap(VfsAwareMapReduceIndex.java:340)
	at com.intellij.util.indexing.impl.MapReduceIndex$IndexUpdateComputable.compute(MapReduceIndex.java:384)
	at com.intellij.util.indexing.impl.MapReduceIndex$IndexUpdateComputable.compute(MapReduceIndex.java:366)
	at com.intellij.openapi.util.Computable.get(Computable.java:18)
	at com.intellij.indexing.composite.CompositeInvertedIndexBase.lambda$mapInputAndPrepareUpdate$3(CompositeInvertedIndexBase.java:73)
	at com.intellij.util.indexing.IndexInfrastructureExtensionUpdateComputation.compute(IndexInfrastructureExtensionUpdateComputation.java:27)
	at com.intellij.util.indexing.IndexInfrastructureExtensionUpdateComputation.compute(IndexInfrastructureExtensionUpdateComputation.java:11)
	at com.intellij.util.indexing.FileBasedIndexImpl.lambda$runUpdateForPersistentData$30(FileBasedIndexImpl.java:1597)
	at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:698)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:646)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computeInNonCancelableSection(CoreProgressManager.java:237)
	at com.intellij.util.indexing.FileBasedIndexImpl.lambda$runUpdateForPersistentData$31(FileBasedIndexImpl.java:1597)
	at com.intellij.util.indexing.StorageBufferingHandler.runUpdate(StorageBufferingHandler.java:22)
	at com.intellij.util.indexing.FileBasedIndexImpl.runUpdateForPersistentData(FileBasedIndexImpl.java:1596)
	at com.intellij.util.indexing.FileBasedIndexImpl.updateSingleIndex(FileBasedIndexImpl.java:1559)
	at com.intellij.util.indexing.FileBasedIndexImpl.lambda$doIndexFileContent$27(FileBasedIndexImpl.java:1408)
	at com.intellij.openapi.fileTypes.impl.FileTypeManagerImpl.freezeFileTypeTemporarilyIn(FileTypeManagerImpl.java:627)
	at com.intellij.util.indexing.FileBasedIndexImpl.doIndexFileContent(FileBasedIndexImpl.java:1376)
	at com.intellij.util.indexing.FileBasedIndexImpl.indexFileContent(FileBasedIndexImpl.java:1330)
	at com.intellij.util.indexing.contentQueue.IndexUpdateRunner.lambda$indexOneFileOfJob$5(IndexUpdateRunner.java:267)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.insideReadAction(NonBlockingReadActionImpl.java:521)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.lambda$attemptComputation$3(NonBlockingReadActionImpl.java:486)
	at com.intellij.openapi.application.impl.ApplicationImpl.tryRunReadAction(ApplicationImpl.java:1084)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.lambda$runInReadActionWithWriteActionPriority$0(ProgressIndicatorUtils.java:75)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runActionAndCancelBeforeWrite(ProgressIndicatorUtils.java:158)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.lambda$runWithWriteActionPriority$1(ProgressIndicatorUtils.java:115)
	at com.intellij.openapi.progress.ProgressManager.lambda$runProcess$0(ProgressManager.java:57)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:188)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$12(CoreProgressManager.java:624)
	at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:698)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:646)
	at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:623)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:66)
	at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:175)
	at com.intellij.openapi.progress.ProgressManager.runProcess(ProgressManager.java:57)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runWithWriteActionPriority(ProgressIndicatorUtils.java:112)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runInReadActionWithWriteActionPriority(ProgressIndicatorUtils.java:75)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.attemptComputation(NonBlockingReadActionImpl.java:486)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.executeSynchronously(NonBlockingReadActionImpl.java:421)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl.executeSynchronously(NonBlockingReadActionImpl.java:186)
	at com.intellij.util.indexing.contentQueue.IndexUpdateRunner.indexOneFileOfJob(IndexUpdateRunner.java:270)
	at com.intellij.util.indexing.contentQueue.IndexUpdateRunner.lambda$indexJobsFairly$4(IndexUpdateRunner.java:210)
	at com.intellij.openapi.progress.impl.ProgressSuspender.executeNonSuspendableSection(ProgressSuspender.java:83)
	at com.intellij.util.indexing.contentQueue.IndexUpdateRunner.indexJobsFairly(IndexUpdateRunner.java:213)
	at com.intellij.util.indexing.contentQueue.IndexUpdateRunner.lambda$doIndexFiles$3(IndexUpdateRunner.java:147)
	at com.intellij.util.concurrency.BoundedTaskExecutor.doRun(BoundedTaskExecutor.java:246)
	at com.intellij.util.concurrency.BoundedTaskExecutor.access$200(BoundedTaskExecutor.java:32)
	at com.intellij.util.concurrency.BoundedTaskExecutor$1.execute(BoundedTaskExecutor.java:225)
	at com.intellij.util.ConcurrencyUtil.runUnderThreadName(ConcurrencyUtil.java:213)
	at com.intellij.util.concurrency.BoundedTaskExecutor$1.run(BoundedTaskExecutor.java:214)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException: Argument for @NotNull parameter 'value' of com/intellij/util/io/EnumeratorStringDescriptor.save must not be null
	at com.intellij.util.io.EnumeratorStringDescriptor.$$$reportNull$$$0(EnumeratorStringDescriptor.java)
	at com.intellij.util.io.EnumeratorStringDescriptor.save(EnumeratorStringDescriptor.java)
	at com.cedricziel.idea.typo3.index.externalizer.StringSetDataExternalizer.save(StringSetDataExternalizer.java:23)
	at com.cedricziel.idea.typo3.index.externalizer.StringSetDataExternalizer.save(StringSetDataExternalizer.java:13)
	at com.intellij.util.indexing.InputMapExternalizer.save(InputMapExternalizer.java:69)
	at com.intellij.util.indexing.InputMapExternalizer.save(InputMapExternalizer.java:19)
	at com.intellij.util.indexing.impl.forward.AbstractForwardIndexAccessor.serializeToByteSeq(AbstractForwardIndexAccessor.java:81)
	at com.intellij.util.indexing.impl.forward.AbstractForwardIndexAccessor.serializeValueToByteSeq(AbstractForwardIndexAccessor.java:69)
	at com.intellij.util.indexing.impl.forward.AbstractForwardIndexAccessor.serializeIndexedData(AbstractForwardIndexAccessor.java:50)
	at com.intellij.util.indexing.impl.forward.AbstractForwardIndexAccessor.serializeIndexedData(AbstractForwardIndexAccessor.java:44)
	at com.intellij.util.indexing.impl.MapReduceIndex.updateForwardIndex(MapReduceIndex.java:276)
	at com.intellij.util.indexing.impl.storage.TransientFileContentIndex.updateForwardIndex(TransientFileContentIndex.java:86)
	at com.intellij.util.indexing.impl.MapReduceIndex.lambda$mapInputAndPrepareUpdate$4(MapReduceIndex.java:261)
	at com.intellij.util.indexing.impl.UpdateData.updateForwardIndex(UpdateData.java:63)
	at com.intellij.util.indexing.impl.MapReduceIndex.updateWithMap(MapReduceIndex.java:348)
	... 57 more
2021-12-22 09:03:07,436 [ 236424]   INFO - il.indexing.FileBasedIndexImpl - Rebuild requested for index com.cedricziel.idea.typo3.index.core_service_map 
com.intellij.util.indexing.StorageException: java.lang.IllegalArgumentException: Argument for @NotNull parameter 'value' of com/intellij/util/io/EnumeratorStringDescriptor.save must not be null
	at com.intellij.util.indexing.impl.MapReduceIndex.updateWithMap(MapReduceIndex.java:355)
	at com.intellij.util.indexing.impl.storage.VfsAwareMapReduceIndex.updateWithMap(VfsAwareMapReduceIndex.java:340)
	at com.intellij.util.indexing.impl.MapReduceIndex$IndexUpdateComputable.compute(MapReduceIndex.java:384)
	at com.intellij.util.indexing.impl.MapReduceIndex$IndexUpdateComputable.compute(MapReduceIndex.java:366)
	at com.intellij.openapi.util.Computable.get(Computable.java:18)
	at com.intellij.indexing.composite.CompositeInvertedIndexBase.lambda$mapInputAndPrepareUpdate$3(CompositeInvertedIndexBase.java:73)
	at com.intellij.util.indexing.IndexInfrastructureExtensionUpdateComputation.compute(IndexInfrastructureExtensionUpdateComputation.java:27)
	at com.intellij.util.indexing.IndexInfrastructureExtensionUpdateComputation.compute(IndexInfrastructureExtensionUpdateComputation.java:11)
	at com.intellij.util.indexing.FileBasedIndexImpl.lambda$runUpdateForPersistentData$30(FileBasedIndexImpl.java:1597)
	at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:698)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:646)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computeInNonCancelableSection(CoreProgressManager.java:237)
	at com.intellij.util.indexing.FileBasedIndexImpl.lambda$runUpdateForPersistentData$31(FileBasedIndexImpl.java:1597)
	at com.intellij.util.indexing.StorageBufferingHandler.runUpdate(StorageBufferingHandler.java:22)
	at com.intellij.util.indexing.FileBasedIndexImpl.runUpdateForPersistentData(FileBasedIndexImpl.java:1596)
	at com.intellij.util.indexing.FileBasedIndexImpl.updateSingleIndex(FileBasedIndexImpl.java:1559)
	at com.intellij.util.indexing.FileBasedIndexImpl.lambda$doIndexFileContent$27(FileBasedIndexImpl.java:1408)
	at com.intellij.openapi.fileTypes.impl.FileTypeManagerImpl.freezeFileTypeTemporarilyIn(FileTypeManagerImpl.java:627)
	at com.intellij.util.indexing.FileBasedIndexImpl.doIndexFileContent(FileBasedIndexImpl.java:1376)
	at com.intellij.util.indexing.FileBasedIndexImpl.indexFileContent(FileBasedIndexImpl.java:1330)
	at com.intellij.util.indexing.contentQueue.IndexUpdateRunner.lambda$indexOneFileOfJob$5(IndexUpdateRunner.java:267)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.insideReadAction(NonBlockingReadActionImpl.java:521)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.lambda$attemptComputation$3(NonBlockingReadActionImpl.java:486)
	at com.intellij.openapi.application.impl.ApplicationImpl.tryRunReadAction(ApplicationImpl.java:1084)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.lambda$runInReadActionWithWriteActionPriority$0(ProgressIndicatorUtils.java:75)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runActionAndCancelBeforeWrite(ProgressIndicatorUtils.java:158)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.lambda$runWithWriteActionPriority$1(ProgressIndicatorUtils.java:115)
	at com.intellij.openapi.progress.ProgressManager.lambda$runProcess$0(ProgressManager.java:57)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$runProcess$2(CoreProgressManager.java:188)
	at com.intellij.openapi.progress.impl.CoreProgressManager.lambda$executeProcessUnderProgress$12(CoreProgressManager.java:624)
	at com.intellij.openapi.progress.impl.CoreProgressManager.registerIndicatorAndRun(CoreProgressManager.java:698)
	at com.intellij.openapi.progress.impl.CoreProgressManager.computeUnderProgress(CoreProgressManager.java:646)
	at com.intellij.openapi.progress.impl.CoreProgressManager.executeProcessUnderProgress(CoreProgressManager.java:623)
	at com.intellij.openapi.progress.impl.ProgressManagerImpl.executeProcessUnderProgress(ProgressManagerImpl.java:66)
	at com.intellij.openapi.progress.impl.CoreProgressManager.runProcess(CoreProgressManager.java:175)
	at com.intellij.openapi.progress.ProgressManager.runProcess(ProgressManager.java:57)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runWithWriteActionPriority(ProgressIndicatorUtils.java:112)
	at com.intellij.openapi.progress.util.ProgressIndicatorUtils.runInReadActionWithWriteActionPriority(ProgressIndicatorUtils.java:75)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.attemptComputation(NonBlockingReadActionImpl.java:486)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl$Submission.executeSynchronously(NonBlockingReadActionImpl.java:421)
	at com.intellij.openapi.application.impl.NonBlockingReadActionImpl.executeSynchronously(NonBlockingReadActionImpl.java:186)
	at com.intellij.util.indexing.contentQueue.IndexUpdateRunner.indexOneFileOfJob(IndexUpdateRunner.java:270)
	at com.intellij.util.indexing.contentQueue.IndexUpdateRunner.lambda$indexJobsFairly$4(IndexUpdateRunner.java:210)
	at com.intellij.openapi.progress.impl.ProgressSuspender.executeNonSuspendableSection(ProgressSuspender.java:83)
	at com.intellij.util.indexing.contentQueue.IndexUpdateRunner.indexJobsFairly(IndexUpdateRunner.java:213)
	at com.intellij.util.indexing.contentQueue.IndexUpdateRunner.lambda$doIndexFiles$3(IndexUpdateRunner.java:147)
	at com.intellij.util.concurrency.BoundedTaskExecutor.doRun(BoundedTaskExecutor.java:246)
	at com.intellij.util.concurrency.BoundedTaskExecutor.access$200(BoundedTaskExecutor.java:32)
	at com.intellij.util.concurrency.BoundedTaskExecutor$1.execute(BoundedTaskExecutor.java:225)
	at com.intellij.util.ConcurrencyUtil.runUnderThreadName(ConcurrencyUtil.java:213)
	at com.intellij.util.concurrency.BoundedTaskExecutor$1.run(BoundedTaskExecutor.java:214)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:668)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1$1.run(Executors.java:665)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/java.util.concurrent.Executors$PrivilegedThreadFactory$1.run(Executors.java:665)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalArgumentException: Argument for @NotNull parameter 'value' of com/intellij/util/io/EnumeratorStringDescriptor.save must not be null
	at com.intellij.util.io.EnumeratorStringDescriptor.$$$reportNull$$$0(EnumeratorStringDescriptor.java)
	at com.intellij.util.io.EnumeratorStringDescriptor.save(EnumeratorStringDescriptor.java)
	at com.cedricziel.idea.typo3.index.externalizer.StringSetDataExternalizer.save(StringSetDataExternalizer.java:23)
	at com.cedricziel.idea.typo3.index.externalizer.StringSetDataExternalizer.save(StringSetDataExternalizer.java:13)
	at com.intellij.util.indexing.InputMapExternalizer.save(InputMapExternalizer.java:69)
	at com.intellij.util.indexing.InputMapExternalizer.save(InputMapExternalizer.java:19)
	at com.intellij.util.indexing.impl.forward.AbstractForwardIndexAccessor.serializeToByteSeq(AbstractForwardIndexAccessor.java:81)
	at com.intellij.util.indexing.impl.forward.AbstractForwardIndexAccessor.serializeValueToByteSeq(AbstractForwardIndexAccessor.java:69)
	at com.intellij.util.indexing.impl.forward.AbstractForwardIndexAccessor.serializeIndexedData(AbstractForwardIndexAccessor.java:50)
	at com.intellij.util.indexing.impl.forward.AbstractForwardIndexAccessor.serializeIndexedData(AbstractForwardIndexAccessor.java:44)
	at com.intellij.util.indexing.impl.MapReduceIndex.updateForwardIndex(MapReduceIndex.java:276)
	at com.intellij.util.indexing.impl.storage.TransientFileContentIndex.updateForwardIndex(TransientFileContentIndex.java:86)
	at com.intellij.util.indexing.impl.MapReduceIndex.lambda$mapInputAndPrepareUpdate$4(MapReduceIndex.java:261)
	at com.intellij.util.indexing.impl.UpdateData.updateForwardIndex(UpdateData.java:63)
	at com.intellij.util.indexing.impl.MapReduceIndex.updateWithMap(MapReduceIndex.java:348)
	... 57 more
	```

@liayn
Copy link
Author

liayn commented Dec 22, 2021

Curiously the issue with the extension oidc seems to be resolved with the updated versions

PhpStorm 2021.3
Build #PS-213.5744.279, built on December 2, 2021

Non-Bundled Plugins:
    com.cedricziel.idea.typo3 (213.7.3)

@sfroemkenjw
Copy link

I can confirm the problem:

PhpStorm 2023.1.2
Build #PS-231.9011.38, built on May 16, 2023

TYPO3 CMS Support: 231.9.2

EXT:oicd 1.2.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants